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From  the  Chairman 
and  Executive  Editor 


The  theme  for  this  edition  of  Defense  Acquisition 
Research  Journal  is  “Challenging  Conventional  Wisdom,” 
for  as  the  articles  in  this  issue  demonstrate,  many  of  the 
most  respected  truisms  in  defense  acquisition  are  not 
nearly  so  clear-cut  as  previously  believed. 


The  first  truism  examined  in  this  edition  is  that  competition  in 
the  defense  market  should  replicate  the  commercial  market  by  reduc¬ 
ing  costs  and  increasing  innovation.  The  article  “DoD  Acquisition— To 
Compete  or  Not  Compete:  The  Placebo  of  Competition,”  by  William 
J.  Levenson,  builds  upon  prior  research  presented  at  the  September 
2012  Defense  Acquisition  University  (DAU)  Research  Symposium, 
“The  Limits  of  Competition  in  Defense  Acquisition”  (http://www.dau. 
mil/research/pages/papers.aspx).  The  author  uses  statistical  model¬ 
ing,  within  a  game  theory  framework  developed  by  Todd  Harrison,  to 
show  that  the  DoD  may  actually  incur  increased  costs  from  competi¬ 
tion.  This  paper  received  the  2013  DAU  Acquisition  Excellence  Award 
for  Outstanding  Acquisition  Paper  at  the  Eisenhower  School,  National 
Defense  University. 


A  second  truism  is  that  the  Operating  and  Support  (O&S)  costs 
account  for  about  70  percent  of  the  total  life-cycle  costs  of  the  aver¬ 
age  weapon  system.  In  “Investigation  into  the  Ratio  of  Operating  and 
Support  Costs  to  Life-Cycle  Costs  for  DoD  Weapon  Systems,”  by  Capt 
Gary  Jones,  USAF,  et  al.,  that  figure  is  shown  to  be  closer  to  55  percent 
of  the  life-cycle  cost,  though  with  substantial  deviation  around  the  mean. 
This  finding  has  significant  implications  for  the  finance  and  budgeting 
of  weapon  systems  programs. 


A  growing  and  pervasive  realization  in  the  world  of  information 
technology  is  that  the  current  cyber  security  mechanisms  such  as 
defense-in-depth,  penetration  test  tools,  and  cyber  test  ranges  may  prove 
insufficient  in  the  face  of  rapidly  evolving  threats.  Thus,  Maj  Bradley  C. 


Panton,  USAF,  and  his  coauthors,  in  their  article  “Strengthening  DoD 
Cyber  Security  with  the  Vulnerability  Market,”  recommend  the  DoD 
adopt  an  economic  strategy  called  the  vulnerability  market  as  a  coop¬ 
erative  means  between  industry  and  the  military  to  not  only  secure,  but 
also  optimize  critical  security  investments. 

Finally,  the  current  emphasis  on  program  costs  as  the  primary  means 
of  optimizing  the  combination  of  cost,  schedule,  and  performance  is  put 
under  a  microscope  in  “A  Conceptual  Framework  for  Defense  Acquisition 
Decision  Makers:  Giving  the  Schedule  Its  Due”  by  Chad  Dacus  and  Col 
Stephen  Hagel,  USAF  (Ret.).  They  argue  that  a  greater  emphasis  on  meet¬ 
ing  schedule  will  provide  a  more  consistent  set  of  outcomes  for  all  three 
criteria. 

The  featured  book  in  this  issue’s  Defense  Acquisition  Professional 
Reading  List  is  John  T.  Kuehn’s  (2008)  Agents  of  INNOVATION:  The 
General  Board  and  the  Design  of  the  Fleet  That  Defeated  the  Japanese 
Navy,  reviewed  by  Robert  G.  “Bob”  Keane. 

On  a  final  note,  I  invite  our  readers  to  note  the  list  of  reviewers  who 
have  so  graciously  given  their  time  and  energy  the  past  year  to  ensure 
that  the  Defense  Acquisition  Research  Journal  maintains  the  highest 
standards  of  editorial  excellence  that  have  made  it  the  world’s  premier 
journal  of  research  on  defense  acquisition. 


( 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 
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DAU  Center  for 
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Research 

Research  Agenda  2014 


The  Defense  Acquisition  Research  Agenda  is  intended  to  make 
researchers  aware  of  the  topics  that  are,  or  should  be,  of  partic¬ 
ular  concern  to  the  broader  defense  acquisition  community 
throughout  the  government,  academic,  and  industrial  sectors. 
The  purpose  of  conducting  research  in  these  areas  is  to  provide 
solid,  empirically  based  findings  to  create  a  broad  body  of  knowl¬ 
edge  that  can  inform  the  development  of  policies,  procedures,  and 
processes  in  defense  acquisition,  and  to  help  shape  the  thought 
leadership  for  the  acquisition  community. 

Each  issue  of  the  Defense  ARJ  will  include  a  different  selection  of 
research  topics  from  the  overall  agenda,  which  is  at:  http://www. 
dau.mil/research/Pages/researchareas.aspx. 

Affordability  and  cost  growth 

•  Define  or  bound  “affordability”  in  the  defense  portfolio.  What  is  it? 
How  will  we  know  if  something  is  affordable  or  unaffordable? 

•  What  means  are  there  (or  can  be  developed)  to  measure,  manage,  and 
control  “affordability”  at  the  program  office  level?  At  the  industry 
level?  How  do  we  determine  their  effectiveness? 

•  What  means  are  there  (or  can  be  developed)  to  measure,  manage, 
and  control  “Should  Cost”  estimates  at  the  Service,  Component, 
program  executive,  program  office,  and  industry  levels?  How  do  we 
determine  their  effectiveness? 

•  What  means  are  there  (or  can  be  developed)  to  evaluate  and  compare 
incentives  for  achieving  “Should  Cost”  at  the  Service,  Component, 
program  executive,  program  office,  and  industry  levels? 


January  2014 


Recent  acquisition  studies  have  noted  the  vast  number  of  programs 
and  projects  that  do  not  make  it  successfully  through  the  acquisition 
system  and  are  subsequently  cancelled.  What  would  systematic 
root  cause  analyses  reveal  about  the  underlying  reasons,  whether 
and  how  these  cancellations  are  detrimental,  and  what  acquisition 
leaders  might  do  to  rectify  problems? 

Do  Joint  programs— at  the  inter-Service  and  international  levels— 
result  in  cost  growth  or  cost  savings  compared  with  single-Service 
(or  single-nation)  acquisition?  What  are  the  specific  mechanisms 
for  cost  savings  or  growth  at  each  stage  of  acquisition?  Do  the  data 
support  “jointness”  across  the  board,  or  only  at  specific  stages  of  a 
program,  e.g.,  only  at  research  and  development  or  only  with  specific 
aspects,  e.g.,  critical  systems  or  logistics? 

Can  we  compare  systems  with  significantly  increased  capability 
developed  in  the  commercial  market  to  DoD-developed  systems  of 
similar  characteristics? 

Is  there  a  misalignment  between  industry  and  the  government 
priorities  that  causes  the  cost  of  such  systems  to  grow  significantly 
faster  than  inflation? 

If  so,  can  we  identify  why  this  misalignment  arises?  What  relation¬ 
ship  (if  any)  does  it  have  to  industry’s  required  focus  on  shareholder 
value  and/or  profit,  versus  the  government’s  charter  to  deliver 
specific  capabilities  for  the  least  total  ownership  costs? 
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DoD  Acquisition-To  Compete 

or  Not  Compete: 

The  Placebo  of  Competition 


\'  William  J.  Levenson 

Commercial  markets  abound  with  examples  of  competi¬ 
tive  forces  providing  reduced  costs  and  increased 
innovation.  However,  the  defense  market  is  materially 
different  from  commercial  markets  in  many  ways,  and 
thus  does  not  respond  in  the  same  way  to  competition. 
This  analysis  examines  a  series  of  outcomes  in  both 
competitive  and  sole-source  acquisition  programs,  using 
a  statistical  model  that  builds  on  a  game  theory  frame¬ 
work  developed  by  Todd  Harrison,  Center  for  Strategic 
and  Budgetary  Assessment.  The  results  show  that  the 
Department  of  Defense  may  actually  incur  increased  costs 
from  competition.  Competition  in  defense  acquisition 
may  not  reduce  costs,  but  may— like  a  placebo— create 
a  powerful  perception  of  cost  control. 
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In  the  never-ending  battle  to  control  the  costs  of  Major  Defense 
Acquisition  Programs  (MDAPs)  in  the  Department  of  Defense  (DoD), 
the  cry  for  competition  can  be  heard  throughout  the  U.S.  Government 
like  an  incantation  to  conjure  the  invisible  hand  of  free  markets  so  aptly 
described  by  Adam  Smith,  the  father  of  modern  economics.  Commercial 
markets  abound  with  examples  of  competitive  forces  providing  reduced 
costs  and  increased  innovation.  Deregulation  and  commercialization  of 
telecommunication  services,  for  example,  broke  up  the  AT&T  monopoly 
and  restored  competition,  resulting  in  the  low-cost,  innovative  prod¬ 
ucts  and  services  now  enjoyed  on  a  global  scale.  Government  officials 
and  politicians  point  to  endless  examples  of  the  powers  of  competition 
in  commercial  markets,  such  as  the  remarkable  consumer  electronics 
available  today  and  the  influence  of  Internet  commerce  that  drives  down 
prices.  Can  the  DoD  harness  these  competitive  forces  to  control  acquisi¬ 
tion  costs  and  provide  innovative  solutions  for  U.S.  defense  needs? 

The  market  for  the  products  and  services  sought  by  the  DoD  differs 
greatly  from  the  free  market  of  the  commercial  economy.  Commercial 
markets  enjoy  a  vast  universe  of  customers,  while  in  most  cases  the 
unique  systems  required  by  the  DoD  make  the  U.S.  Government  the 
sole  customer  and  regulator.  The  companies  of  the  military-industrial 
complex  of  the  1950s  have  consolidated  and  specialized  over  the  past 
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two  decades,  strengthening  oligopolies  and  creating  monopolies,  thereby 
limiting  opportunities  for  competition.  This  article  employs  a  simplified 
statistical  model  to  examine  various  characteristics  of  competition  in 
defense  markets  and  to  provide  insight  for  acquisition  professionals  and 
policymakers.  The  results  indicate  that  cost  savings,  when  they  occur, 
often  come  with  adverse  side  effects  on  budget  planning  and  industry 
health.  Uncertainties  in  the  competitive  bid  process  can  cause  large  cost 
variations,  overwhelming  the  savings  from  competitive  pressures  on 
profit  margins.  Innovation  introduced  by  competition  can  reduce  costs, 
but  innovation  can  be  difficult  to  distinguish  from  overly  optimistic  cost 
estimates,  particularly  when  sellers  have  to  set  prices  before  product 
development  and  production.  Contrary  to  expectations,  competition  may 
actually  increase  costs  relative  to  sole-source  procurement. 

Background 

The  desire  for  competition  has  a  long  history  in  federal  acquisi¬ 
tion.  In  1809,  Congress  passed  the  first  law  addressing  the  question, 
stating  the  preference  for  “formal  advertising”  for  procurement  con¬ 
tracts.  Subsequent  legislation  periodically  relaxed  and  strengthened 
requirements  for  competition  in  response  to  various  wartime  and 
peacetime  demands.  The  Competition  in  Contracting  Act  (CICA)  of 
1984  laid  the  foundation  for  today’s  regulations,  requiring  “full  and 
open  competition  through  the  use  of  competitive  procedures”  (Manuel, 
2011,  p.  4).  Subsequent  legislation  has  amended  CICA  and  allows  for 
many  alternatives  to  competition  under  specific  conditions  (Manuel, 
2011).  Most  recently,  in  response  to  increased  DoD  acquisition  costs  and 
growing  budget  pressures,  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  Frank  Kendall  reemphasized  competition  to 
control  and  reduce  cost  (Kendall,  2012).  Belying  the  recent  emphasis 
on  competition  to  mitigate  cost  challenges,  a  recent  study  implies  that 
competition  will  often  increase  the  cost  of  acquisition. 

Todd  Harrison  of  the  Center  for  Strategic  and  Budgetary  Assessment 
performed  a  game  theory  analysis  of  two  equal  competitors  bidding  on 
a  hypothetical  acquisition  program  (Harrison,  2012).  He  assumed  each 
competitor  had  perfect  knowledge  of  the  development  and  production 
costs,  and  each  could  bid  either  a  10  percent  profit,  zero  profit,  or  a  10 
percent  loss  in  any  given  round  of  competition  (the  analysis  allowed  for 
a  loss  in  a  round  of  bidding  as  a  strategy  to  win  future  rounds,  but  recog¬ 
nized  that  a  competitor  would  not  accept  a  loss  over  the  entire  program). 
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Harrison’s  analysis  shows  that  sole-source  procurement  provides  the 
lowest  range  of  potential  costs  regardless  of  the  number  of  rounds  or 
the  award  split  used  in  a  competitive  acquisition  strategy  (Figure  1). 
Competition  produces  higher  costs  in  this  analysis  because  each  com¬ 
petitor  incurs  duplicative  development  costs,  and  neither  competitor 
can  realize  the  full  cost  benefits  of  a  typical  production  learning  curve. 
Harrison’s  game  theory  model  of  competition  examines  the  bidding 
behavior  of  two  equal  competitors,  but  it  does  not  address  character¬ 
istics  that  differentiate  competitors  or  recognize  imperfect  knowledge 
about  the  costs  of  development  and  production.  A  statistical-modeling 
approach  can  explore  these  characteristics. 

FIGURE  1.  MULTIPLE  ROUNDS  OF  COMPETITION 


Statistical  Model  and  Analysis  Methodology 

To  investigate  how  the  outcomes  of  sole-source  procurement 
compare  to  competitive  procurement  would  require  researchers  to 
execute  duplicate  acquisition  programs  as  both  a  competitive  and  sole- 
source  procurement  and  compare  the  results.  To  gain  meaningful  data, 
researchers  would  have  to  conduct  this  experiment  many  times.  Of 
course,  such  a  real-world  trial  would  be  virtually  impossible.  Instead, 
this  comparison  can  be  made  by  using  a  statistical  model  of  bidding  and 
program  execution,  with  comparison  of  the  results  of  multiple  trials  in 
a  Monte  Carlo  analysis  (Figure  2). 
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FIGURE  2.  FLOW  CHART  OF  COMPETITION  MODEL 
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The  statistical  model  attempts  to  capture  several  key  characteris¬ 
tics  that  affect  the  final  price  of  an  acquisition  program,  including  profit 
sought  by  supplier,  the  accuracy  of  estimates  used  to  produce  supplier 
bids,  innovations  that  reduce  the  true  cost  for  one  competitor,  and  the 
amount  of  prior  experience  each  supplier  has  in  developing  and  producing 
similar  products.  Whereas  Harrison’s  analysis  assumed  the  supplier  bids 
reflect  the  true  program  cost,  this  approach  evaluates  the  effects  on  the 
final  program  costs,  which  often  vary  substantially  from  initial  proposals. 

The  analysis  methodology  starts  with  the  same  hypothetical  acquisi¬ 
tion  program  used  by  Harrison,  which  assumed  a  $2,000  development 
phase,  a  100-unit  production  run,  and  a  $1,000  cost  for  the  first  unit. 
The  cost  of  subsequent  units  benefits  from  a  learning  curve  defined  as 
Cn=C  p  nlog2  °'85 ,  where  Cn  is  the  cost  of  the  nth  unit,  and  C  p  is  the  cost  of 
the  first  unit  (for  a  $1,000  first-unit  cost  and  100-unit  purchase,  the  aver¬ 
age  cost  becomes  $435).  To  evaluate  the  effects  of  competition,  the  model 
assumes  that  two  competitors  bid  on  the  development  and  production 
phases,  and  the  lowest  bid  wins.  The  final  cost  of  the  competition  is  then 
compared  to  that  of  sole-source  procurement.  In  cases  that  examine  the 
effects  of  random  variables,  the  analysis  uses  random  values  for  both  the 
competitive  and  the  sole-source  procurement,  compares  final  buyer  cost 
for  each,  and  repeats  the  process  1,000  times  to  obtain  statistical  data  on 
the  cost  difference  to  the  buyer,  as  well  as  on  several  other  parameters. 

Analysis  Results 

The  following  results  illustrate  the  effects  of  each  competition 
characteristic.  Examining  each  effect  independently  provides  insight 
that  acquisition  officials  and  policymakers  can  use  to  assess  the  com¬ 
petitive  environment  for  a  product,  consider  whether  to  emphasize 
competition,  evaluate  competitors  and  their  proposals,  and  establish 
expectations  for  the  results.  The  analysis  begins  with  the  most  basic 
aspect  of  competition:  the  pressure  on  suppliers  to  trim  profit  margins 
to  win  a  competition. 

Case  1:  Competitive  Pressure  on  Profit  Margin 

To  remain  consistent  with  Harrison’s  results,  the  analysis  of  reduced 
profit  margins  assumes  the  sole-source  supplier  requires  a  profit  of 
10  percent,  resulting  in  a  total  cost  to  the  buyer  of  $50,050.  Since  a 
sole-source  provider  feels  no  pressure  to  trim  the  profit  margin,  the  anal¬ 
ysis  holds  the  10  percent  profit  constant  for  this  case.  The  competitors, 
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however,  will  feel  pressure  to  lower  their  profit  margin  to  win  the  com¬ 
petition.  Business  conditions  for  each  competitor— such  as  weighted 
average  cost  of  capital,  manufacturing  capacity,  and  the  expectations  of 
shareholders— will  influence  how  much  profit  each  competitor  requires. 
The  model  will  treat  the  profit  margin  contained  in  each  competitor’s 
bid  as  a  random  variable  with  a  mean  and  standard  deviation.  Figure  3 
shows  the  average,  1-sigma  (one  standard  deviation)  variation,  and  the 
range  from  5th  percentile  to  95th  percentile,  for  the  savings  the  buyer  can 
expect  as  the  mean  bid  varies  from  5  to  10  percent,  assuming  a  bid  stan¬ 
dard  deviation  of  2  percent  profit.  Not  surprisingly,  sellers’  reductions  in 
acceptable  profit  margins  lead  directly  to  savings  for  the  buyer.  Note  that 
the  average  saving  exceeds  the  simple  difference  in  mean  profit  margin 
between  sole-source  and  competitive  bidding  because  the  competitive 
process  selects  the  lower  of  the  two  bidders.  Thus,  for  example,  a  mean 
bid  profit  margin  of  7  percent  saves  the  buyer  not  only  the  3  percent  dif¬ 
ference  between  7  percent  and  the  sole-source  bid  of  10  percent,  but  an 
additional  saving  occurs  by  selecting  the  lowest  of  two  bidders,  resulting 
in  an  average  3.7  percent  saving  to  the  buyer.  If  more  suppliers  enter  the 
competition,  the  saving  will  marginally  improve. 


FIGURE  3.  EFFECT  OF  PROFIT  MARGIN  PRESSURES 


Note.  Competitive  pressures  can  force  sellers  to  reduce  profit  margins,  resulting  directly 
in  buyer  savings  relative  to  sole-source  procurement. 
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This  somewhat  obvious  first  case  illustrates  two  characteristics 
of  competition  relative  to  sole-source  procurement.  First,  competitive 
pressure  on  profit  margins  alone  provides  a  relatively  modest  saving  to 
the  buyer.  The  defense  industry  generally  has  the  lowest  profit  margins 
among  its  peers  in  other  industries,  ranging  between  5  percent  and 
10  percent  for  the  period  from  1989  to  2006  (Arnold,  Harmon,  Tyson, 
Fasana,  &  Wait,  2009,  p.  50).  Starting  at  these  low  profit  margins  pro¬ 
vides  limited  opportunity  to  shave  margins  further. 

Second,  a  seller’s  willingness  to  reduce  its  profit  margin  signifi¬ 
cantly  to  win  a  competition  may  indicate  a  struggling  business.  For 
example,  executives  interviewed  by  the  Space  Industry  Study  Group 
at  the  Eisenhower  School  for  National  Security  and  Resource  Strategy 
recently  noted  that  each  competition  seems  to  have  at  least  one  desperate 
bidder,  due  to  declining  federal  budgets.  While  these  low  bids  promise 
savings  for  the  government,  acquisition  officials  should  carefully  con¬ 
sider  whether  the  low  bidder  can  survive  unexpected  cost  increases  and 
reliably  deliver  the  final  product. 

Like  Harrison’s  analysis,  this  first  case  assumes  that  the  competitors 
have  perfect  knowledge  of  the  development  and  production  costs  for  the 
proposed  product.  Most  MDAPs,  however,  do  not  enjoy  perfect  knowledge 
of  these  future  costs. 

Case  2:  Bidding  Accuracy 

In  addition  to  monopsonistic  and  oligopolistic  conditions,  the  defense 
acquisition  market  differs  from  the  commercial  markets  in  another  fun¬ 
damental  way.  The  defense  industry  faces  significant  uncertainty  in  its 
costs  at  the  time  it  sets  the  price  for  its  products.  In  atypical  commercial 
market,  a  seller  offers  a  product  for  sale  after  completing  development 
and  an  initial  production  run.  At  that  point,  the  seller  understands  its 
costs,  can  evaluate  the  demand,  and  can  set  the  price  and  production 
rate  to  maximize  profits  and  effectively  compete  against  other  sellers. 
In  most  MDAPs,  the  buyer  asks  the  seller  to  set  the  price  in  advance. 
This  analysis  case  will  examine  two  effects  of  imperfect  cost  knowledge. 
First,  the  analysis  assumes  that  both  the  sole-source  provider  and  the 
two  competitors  have  the  same  inaccuracy  in  assessing  the  future  cost  of 
the  program.  Then  the  analysis  will  evaluate  the  results  if  one  competitor 
underestimates  the  true  costs. 
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Case  2a:  Equal  bidding  inaccuracy.  To  assess  the  effects  of  bid¬ 
ding  inaccuracy  on  the  outcomes  of  competition,  the  analysis  assumes 
the  true  costs  of  development  and  production  remain  the  same  as  Case 
1,  but  that  each  seller  has  independent  inaccuracy  in  estimating  them.  A 
random  factor  with  a  Gaussian  distribution  represented  by  its  standard 
deviation  (1-sigma)  value  will  signify  this  uncertainty  in  the  model. 
The  bids  of  the  sole-source  provider  and  the  two  competitors  will  vary 
independently  based  on  the  same  standard  deviation. 

This  Case  2a  scenario  simulates  a  condition  in  which  any  seller  will 
likely  incur  the  same  cost  in  the  end,  but  be  unable  to  estimate  accurately 
the  cost  changes  that  might  occur  during  the  development  and  produc¬ 
tion  learning  curve.  In  this  case,  the  contract  type  becomes  relevant  to 
the  outcome.  The  analysis  will  evaluate  both  a  Fixed  Price  (FP)  and  Cost 
Reimbursable  (Cost-Plus,  or  CP)  contract.  For  an  FP  contract,  the  buyer 
pays  an  agreed  price  regardless  of  the  actual  cost  incurred  by  the  seller, 
and  therefore  the  seller  could  earn  a  profit  higher  or  lower  than  the  bid 
contains.  For  a  CP  contract,  the  buyer  agrees  to  pay  a  fee,  representing 
the  seller’s  profit,  based  on  a  percentage  of  the  initially  estimated  costs, 
and  the  buyer  also  pays  the  actual  costs  of  development  and  production. 

Figure  4  shows  the  cost  savings  that  competition  provides  rela¬ 
tive  to  sole-source  procurement  for  an  FP  contract.  The  inaccuracy  of 
the  initial  bid  for  an  FP  contract  competition,  on  average,  reduces  the 
buyer’s  final  cost  relative  to  a  sole-source  award.  The  result  has  high 
variability;  however,  competition  actually  results  in  higher  cost  relative 
to  sole-source  procurement  in  about  one-third  of  the  simulations.  In  this 
case  as  in  Case  1,  the  buyer  benefits  at  the  seller’s  expense,  but  since,  by 
definition,  the  seller  cannot  predict  its  bid  inaccuracies,  the  potential 
impact  to  the  seller’s  profit  (or  loss)  can  be  much  greater.  Figure  5  shows 
how  inaccurate  bidding  affects  the  competitive  seller’s  profits.  While  FP 
contracts  place  this  risk  on  the  seller,  two  issues  could  arise  for  acqui¬ 
sition  programs:  (a)  if  the  seller  incurs  too  much  loss,  it  could  become 
unable  to  complete  the  contract;  and  (b)  if  the  seller  incurs  too  much 
gain,  it  could  draw  the  attention  of  regulators  always  on  the  lookout  for 
excessive  industry  profits.  For  the  sole-source  seller,  the  average  profit 
does  not  decline,  but  it  has  somewhat  more  variability.  The  competitive 
seller  suffers  more  on  average  because  the  competition  selects  the  bidder 
who  underestimates  the  costs  the  most. 


Defense  ARJ,  January  2014,  Vol.  21  No.  1 : 416-440 


424 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


FIGURE  4.  BID  ACCURACY-FIXED  PRICE  CONTRACT 
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Note.  As  the  inaccuracy  of  the  initial  bid  increases  for  a  Fixed  Price  contract,  competition, 
on  average,  reduces  the  buyer’s  final  cost  relative  to  a  sole-source  award;  however,  the 
result  has  high  variability.  Competition  actually  increases  cost  relative  to  sole  source  in 
about  one-third  of  the  simulations. 


FIGURE  5.  BID  ACCURACY-FIXED  PRICE  CONTRACT 


Note.  Bid  inaccuracy  can  severely  impact  seller  profits  on  a  Fixed  Price  contract. 
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FIGURE  6.  BID  ACCURACY-COST  REIMBURSABLE  CONTRACT 


Note.  Minimal  savings  from  competition  for  a  Cost  Reimbursable  contract:  the  initial  bid 
only  affects  the  fee  paid  to  the  seller  as  profit,  so  bidding  accuracy  influences  the  final 
cost  only  slightly. 
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FIGURE  7.  BID  ACCURACY-COST  REIMBURSABLE  CONTRACT 

25.0% 

20.0% 

15.0% 

„  10'0% 

|  5.0% 

-U 

I  0.0% 

“  -5.0% 

-10.0% 

-15.0% 

-20.0% 


|  1-Sigma  Range 
Mean 


5th  Percentile 


BID  UNCERTAINTY  (1-SIGMA) 


Note.  Bid  uncertainty  leads  to  cost  overruns  on  Cost  Reimbursable  contracts. 
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Figure  6  shows  that  bidding  inaccuracy  for  a  CP  contract  does  not 
favor  either  sole-source  or  competitive  procurement.  In  either  case, 
the  initial  bid  affects  only  the  fee  paid  to  the  seller  as  profit,  so  bidding 
accuracy  influences  the  final  cost  to  the  buyer  only  slightly.  The  effect 
for  a  CP  contract  reveals  itself  in  the  potential  cost  overruns  relative  to 
initial  bids.  Figure  7  shows  the  cost  overrun  statistics  for  the  competitive 
seller  in  a  CP  contract.  The  average  cost  overrun  increases  for  a  competi¬ 
tive  seller,  but  stays  near  zero  for  the  sole-source  seller  because,  like  the 
effect  on  profit  under  an  FP  contract,  the  competition  selects  the  seller 
that  underestimates  the  cost  the  most.  Therefore,  competition  for  a  CP 
contract  will  tend  to  increase  the  likelihood  of  cost  overruns,  but  not 
significantly  reduce  the  final  cost  to  the  buyer.  In  addition  to  random 
bidding  inaccuracy,  sellers  may  also  have  abias  toward  underestimating 
development  and  production  costs,  which  Case  2b  examines. 

Case  2b:  One  competitor  underestimates  costs.  In  Case  2a,  the 
sole-source  seller  and  both  competitive  sellers  have  uncertainty  about 
the  final  costs,  but  on  average  their  bids  reflect  the  true  cost.  Often  when 
forecasting  cost,  sellers  will  not  only  have  inaccuracy,  but  abias  toward 
underestimation.  Especially  in  the  face  of  competition,  a  seller  will  tend 
to  estimate  costs  optimistically.  To  represent  this  effect,  the  model  can 
add  a  bias  to  the  bids,  while  keeping  the  true  costs  of  development  and 
production  the  same  as  previous  cases.  Adding  identical  bias  to  the  sole- 
source  seller  and  both  competitive  sellers  would  merely  shift  the  results 
of  Case  2a.  Instead,  Case  2b  assumes  that  one  competitive  bidder  has 
less  experience  developing  and  producing  the  product  than  either  the 
other  competitor  or  the  sole-source  seller.  For  the  less  experienced  seller, 
the  analysis  assumes  a  bidding  inaccuracy  of  10  percent  (representing 
1-sigma)  and  a  bias  toward  underestimating  the  true  cost.  Meanwhile, 
both  the  sole-source  seller  and  the  other  competitor  will  have  a  some¬ 
what  lower  bidding  inaccuracy  of  5  percent  without  a  bias. 

Figures  8  and  9  show  the  results  for  an  FP  contract  as  the  bias  varies 
from  0  to  10  percent.  As  with  Case  2a,  the  buyer  benefits  from  a  lower 
cost  at  the  expense  of  the  seller’s  profit.  An  inexperienced  competitor 
underestimating  the  final  cost  amplifies  both  effects.  The  competitive 
selection  favors  the  inexperienced  seller  with  the  lower  bid,  creating 
more  buyer  savings  relative  to  a  sole-source  seller,  but  increasing  the 
concern  that  the  seller  will  incur  intolerably  low  profit,  or  even  losses. 
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FIGURE  8.  UNDERBID-FIXED  PRICE  CONTRACT 


Note.  A  less  experienced  competitor  will  tend  to  have  less  accurate  estimates  and  may 
underestimate  final  costs.  As  a  competitor's  underestimate  increases  for  a  Fixed  Price 
contract,  the  buyer  enjoys  a  more  favorable  price  than  a  more  accurate  bid  from  a  sole- 
source  seller. 


FIGURE  9 
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Note.  A  less  experienced  competitor  will  suffer  reduced  profits,  or  even  losses,  due 
to  underestimated  costs.  Since  competition  will  favor  the  lower  bidder,  competitive 
procurements  will  make  seller  losses  more  likely  than  sole-source  procurements. 
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FIGURE  10.  UNDERBID-COST  REIMBURSABLE  CONTRACT 
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Note.  For  a  Cost  Reimbursable  contract,  bid  inaccuracy  and  bid  underestimation 
provide  little  advantage  for  competition  relative  to  sole-source  procurement  because 
the  final  price  to  the  buyer  includes  any  cost  variation  from  the  bids.  Only  the  fee,  or 
profit,  paid  to  the  seller  varies— and  only  slightly. 


FIGURE  11.  UNDERBID-COST  REIMBURSABLE  CONTRACT 
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Note.  Bid  inaccuracy  and  bias  toward  underestimation  can  amplify  cost  overruns  in  Cost 
Reimbursable  when  comparing  sole-source  procurement  to  competitive  procurement 
because  competitive  selection  will  favor  the  underestimated  bid. 
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Figures  10  and  11  show  the  results  for  a  CP  contract.  As  with  Case 
2a,  the  effects  of  bid  inaccuracy  and  bias  in  competitive  procurement 
provide  little  savings  to  the  buyer  relative  to  sole-source  procurement. 
Bid  underestimation,  however,  has  significant  effects  on  cost  over¬ 
runs,  which  can  threaten  the  program  and  disrupt  future  planning  if 
not  taken  into  account.  Competitive  procurement  increases  the  effect 
relative  to  sole-source  procurement  because  the  competitive  selection 
process  favors  the  underestimated  bid. 

Cases  2a  and  2b  demonstrate  that  bid  inaccuracy  and  bias  can 
indeed  lower  the  final  cost  to  the  buyer,  but  at  the  expense  of  the  seller’s 
profit  on  FP  contracts,  or  of  more  likely  cost  overruns  on  CP  contracts, 
both  of  which  can  put  the  entire  program  at  risk.  The  Navy’s  A-12  acqui¬ 
sition  program,  which  was  cancelled  in  1991,  provides  an  example. 

In  1984,  a  partnership  between  McDonnell  Douglas  and  General 
Dynamics  won  the  FP  contract  for  the  A-12  bomber  with  a  bid  of 
$4.8  billion,  substantially  beating  the  $5.9  billion  bid  of  the  partner¬ 
ship  of  Northrop  and  Grumman  (Wilson  &  Carlson,  1995).  Northrop 
and  Grumman  had  prior  experience  with  the  key  stealth  technology 
required  for  the  contract,  while  the  winning  bidder  did  not.  Cost  over¬ 
runs  and  schedule  delays  began  almost  immediately  after  the  contract 
was  signed.  Although  it  was  an  FP  contract,  the  sellers  could  not  absorb 
the  cost  increases  and  filed  a  Request  for  Equitable  Adjustment  for  a 
price  increase  of  $1.47  billion,  implying  the  losing  bid  of  Northrop  and 
Grumman  more  closely  estimated  the  true  costs.  The  DoD  cancelled 
the  program  in  1991  due  to  these  cost  increases,  spawning  a  lawsuit 
from  the  sellers  alleging,  in  part,  that  the  DoD  had  knowledge  that  the 
requirements  were  “unattainable”  (Wilson  &  Carlson,  1995,  p.  W.10). 
The  use  of  an  FP  contract  with  an  underinformed  competitor  produced 
a  contract  price  that  seemed  like  a  good  deal  at  the  time,  but  ultimately 
resulted  in  the  cancellation  of  the  program  and  spawned  a  lawsuit  that 
continues  today,  two  decades  later. 

Thus  far,  the  analysis  has  evaluated  differences  in  competitor  bid¬ 
ding  characteristics  while  keeping  the  actual  cost  of  development  and 
production  fixed.  The  next  case  examines  the  effects  of  competition 
where  the  sellers  have  differing  design  solutions  or  production  pro¬ 
cesses  that  change  the  actual  cost. 
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Case  3:  Innovation 

Innovation  in  design  or  production  can  create  differences  among 
sellers,  resulting  in  true  cost  savings  in  the  final  product.  Utilizing 
proven  technology  in  a  design,  for  example,  might  produce  a  less  expen¬ 
sive  product  than  a  design  that  requires  new  technology  development. 
Advanced  manufacturing  techniques  or  automated  assembly  could  lower 
the  production  costs  compared  to  existing  manufacturing  methods  or 
manual  assembly. 

In  Case  3,  the  analysis  assumes  that  all  of  the  sellers  have  a  5  per¬ 
cent  standard  deviation  in  the  actual  cost  of  their  product  and  that  one 
competitive  seller  has  reduced  the  cost  of  production  by  up  to  10  percent. 
Because  innovation  likely  implies  higher  development  costs,  the  analysis 
assumes  that  the  seller  with  the  lower  production  cost  will  also  incur  a 
20  percent  higher  development  cost,  which  as  it  turns  out  does  not  sig¬ 
nificantly  affect  the  conclusions. 


FIGURE  12.  SAVINGS  FROM  COMPETITION  AND  INNOVATION 
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Note.  Innovation  in  design  or  manufacturing  can  change  the  true  cost.  As  the  production 
savings  of  one  competitive  seller  improve,  the  buyer  benefits  from  competition  relative 
to  a  sole-source  seller  without  innovation. 


Since  all  of  the  sellers  in  this  case  have  accurate  bid  estimates  (per¬ 
fect  knowledge  of  future  costs),  FP  and  CP  contracts  give  the  same 
results  when  comparing  the  cost  of  the  competitive  procurement  to  the 
sole-source  procurement.  Figure  12  shows  the  benefits  that  competitive 
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procurement  provides  over  sole-source  procurement  when  one  com¬ 
petitor  includes  cost-saving  innovations.  Although  the  analysis  assumes 
higher  development  costs,  the  savings  during  production  and  the  com¬ 
petitive  selection  of  the  lowest  cost  seller  overcome  that  disadvantage. 
Therefore,  a  buyer  should  favor  competitive  procurement  over  a  sole- 
source  procurement  if  any  seller  appears  to  offer  innovative  solutions 
that  lower  costs.  The  Evolved  Expendable  Launch  Vehicle  (EELV)  pro¬ 
gram  may  provide  an  example. 

Recently,  the  Air  Force  reinvigorated  competition  for  space  launch 
contracts  under  the  EELV  program  (Leone,  2012).  New  entrants  into  this 
market,  such  as  Space  Technologies,  Inc.  (SpaceX),  claim  that  they  can 
provide  launch  services  at  lower  costs  than  the  current  provider,  United 
Launch  Alliance,  LLC.  SpaceX  expects  its  vertically  integrated  supply 
chain  and  a  more  efficient  design  will  provide  cost  savings  (Chaikin, 
2012).  Case  3  of  this  analysis  indicates  that  the  Air  Force  may  reap 
true  cost  savings  from  this  innovation-based  competition.  It  remains 
unclear,  however,  whether  this  innovation  can  actually  provide  cost 
savings,  or  whether  SpaceX  has  underestimated  its  cost  as  in  Case 
2b.  While  innovation  may  offer  cost  improvements,  incumbent  sellers 
in  a  defense  market  may  have  cost  advantages  over  their  competition 
due  to  their  experience  developing  and  producing  similar  products. 

Case  4:  Incumbent  Advantages 

In  many  defense  markets,  corporate  consolidation  has  created  oli¬ 
gopolies  and  monopolies,  with  a  few  companies  competing  for  a  limited 
number  of  defense  programs.  Corporate  consolidation  in  many  markets 
has  also  created  strong  incumbents  who  have  much  greater  capabilities 
and  experience  producing  their  products  than  potential  competitors. 
Two  factors  complicate  the  introduction  of  competition  into  a  market 
with  a  strong  incumbent.  First,  an  incumbent’s  existing  expertise  and 
infrastructure  provide  a  cost  advantage  over  a  new  competitor.  A  new 
competitor  may  have  to  invest  more  in  infrastructure  and  development 
to  catch  up. 

Second,  incumbents  will  have  a  better  ability  to  forecast  the  costs 
of  a  new  product  in  their  area  of  expertise.  A  new  entrant,  with  less 
understanding  of  the  challenges  and  complexities  of  a  particular  prod¬ 
uct,  could  easily  underestimate  the  cost  of  development.  In  competitive 
procurement,  these  two  factors  can  conspire  to  create  the  illusion  of 
equal  competitors.  The  incumbent  will  provide  an  accurate  bid  that 
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reflects  its  actual  cost  advantage,  while  the  new  entrant  will  likely  have 
higher  development  costs  and  start  higher  on  the  production  learning 
curve,  but  will  underestimate  the  costs  in  its  bid  either  out  of  ignorance 
or  from  competitive  pressures.  Both  may  decide  to  trim  profit  margins 
to  better  compete. 

To  represent  this  situation,  the  analysis  for  Case  4  assumes  the 
following: 

•  The  incumbent  has  a  random  bid  inaccuracy  of  5  percent 
(1-sigma)  without  a  bias. 

•  The  new  entrant  has  a  random  bid  inaccuracy  of  10  percent 
(1-sigma)  with  an  average  underestimate  of  15  percent. 

•  The  incumbent  starts  lower  on  the  production  learning 
curve,  making  its  production  costs  15  percent  lower  than 
the  new  entrant. 

•  The  new  entrant  has  20  percent  higher  development  costs. 

•  The  incumbent  represents  the  sole-source  seller,  with  no 
pressure  to  reduce  profit  margins  from  10  percent. 

•  The  incumbent  and  the  new  entrant  compete  and  bid  an  8 
percent  mean  profit  with  a  standard  deviation  of  1  percent 
profit. 

Within  these  parameters,  the  model  produces  two  families  of  poten¬ 
tial  outcomes  for  a  CP  contract,  as  shown  in  Figure  13.  In  cases  where 
the  incumbent  wins  the  competition,  the  results  reflect  a  slight  (approxi¬ 
mately  2  percent)  cost  savings  relative  to  sole-source  procurement, 
primarily  due  to  the  modeled  competitive  pressure  to  reduce  profit 
margins.  When  the  new  entrant  wins  the  competition,  however,  its  cost 
disadvantages  manifest  themselves  with  substantially  higher  costs  to 
the  buyer  relative  to  sole-source  procurement  from  the  incumbent.  This 
illustrates  the  danger  of  forcing  competition  between  a  strong  incumbent 
and  a  relatively  inexperienced  new  entrant. 
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FIGURE  13.  COST  SAVINGS  (OR  INCREASE)  FROM  COMPETITION 


COST  SAVINGS  (OR  INCREASE)  FROM  COMPETITION 


Note.  Histogram  of  outcomes  from  a  competition  for  a  Cost  Reimbursable  contract 
between  an  incumbent  and  a  new  entrant  as  compared  to  a  sole-source  procurement 
from  the  incumbent.  When  the  incumbent  wins  the  competition,  competitive  pressures 
on  profit  margin  result  in  the  slight  cost  savings  relative  to  sole-source  procurement. 
When  the  new  entrant  wins,  its  higher  cost  of  development  and  production  results  in  a 
substantial  cost  increase  relative  to  a  sole-source  procurement. 


The  competition  for  the  Future  Imaging  Architecture  contract  in 
1998-99  provides  an  example  of  these  Case  4  characteristics.  To  the 
surprise  of  many,  the  National  Reconnaissance  Office  decided  to  open  its 
next-generation  imaging  satellites  to  new  competition,  despite  Lockheed 
Martin’s  four-decade  heritage  as  the  sole  provider  of  this  technology. 
Boeing,  looking  to  diversify  its  business,  entered  the  competition  and 
won,  beating  Lockheed  Martin  with  a  proposal  determined  to  be  cheaper 
and  more  innovative  (Taubman,  2007). 

Lockheed  had  a  strong  incumbent  position,  with  more  than  $30  bil¬ 
lion  invested  by  the  government  into  its  capabilities  to  produce  imaging 
satellites.  Boeing  had  little  experience  and  vastly  underestimated  the 
cost,  likely  from  a  combination  of  ignorance  and  a  desire  to  meet  the 
government’s  price  targets.  As  Boeing  realized  development  challenges, 
the  subsequent  cost  growth  ultimately  caused  the  cancellation  of  the 
program  at  a  loss  of  more  than  $4  billion  (Taubman,  2007). 
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Summary 

This  study  analyzed  four  aspects  of  competition  relevant  to  the  DoD’s 
effort  to  control  and  reduce  the  cost  of  MDAPs:  Case  1,  Competitive 
Pressure  on  Profit  Margin;  Case  2,  Bidding  Accuracy;  Case  3,  Innovation; 
and  Case  4,  Incumbent  Advantages.  The  results  call  into  question  the 
justification  of  promoting  competition  on  the  basis  of  cost,  and  offer 
acquisition  officials  and  policymakers  insights  into  the  outcomes  they 
can  expect  from  competition: 

•  Competition  may  pressure  sellers  to  trim  profit  margins,  but 
the  defense  industry  already  operates  with  low  margins  rel¬ 
ative  to  its  peers.  Cost  variation  from  other  characteristics 
of  competitive  procurement,  such  as  bidding  uncertainty, 
overwhelms  the  modest  cost  reduction  available  from  profit 
margins.  When  weighing  competitive  procurement  against 
sole-source  procurement,  DoD  acquisition  officials  should 
look  beyond  mere  pricing  pressures  for  benefits  and  risks. 

•  Unlike  commercial  markets,  markets  for  MDAPs  usually 
require  sellers  to  set  prices  before  they  know  the  develop¬ 
ment  and  production  costs.  DoD  acquisition  officials  should 
consider  the  potential  inaccuracy  of  cost  estimates  when 
selecting  contract  type  and  recognize  that  competition 
increases  the  likelihood  and  severity  of  seller  losses  or  cost 
overruns  that  could  threaten  program  completion. 

°  For  FP  contracts,  inaccuracies  in  cost  predictions 
have  a  more  powerful  and  unpredictable  influence 
on  cost  outcomes  than  the  competitive  pressures  on 
profit  margins,  resulting  in  more  cost  than  sole-source 
procurement  in  one-third  of  the  cases.  Savings  to  the 
buyer  come  at  the  expense  of  seller  profits,  which  could 
threaten  the  ability  of  the  seller  to  complete  the  pro¬ 
gram  as  profits  diminish  or  losses  increase. 

°  Inaccuracies  in  cost  predictions  for  CP  contracts  have 
little  effect  on  final  cost  to  the  buyer,  but  can  signifi¬ 
cantly  increase  cost  overruns,  threatening  completion 
of  the  program  and  complicating  future  planning. 


435 


Defense ARJ,  January  2014,  Vol.  21  No.  1: 416-440 


DoD  Acquisition  —  To  Compete  or  Not  Compete:  The  Placebo  of  Competition 


Innovation  in  design  or  production  offers  the  best  rationale 
to  promote  competition  over  sole-source  procurement  to 
reduce  costs  to  the  buyer.  Competitors  that  offer  cost  sav¬ 
ings  through  more  efficient  design  or  advanced  production 
processes  can  directly  influence  the  final  cost  to  the  buyer. 
DoD  acquisition  officials  face  the  challenge,  however,  of 
distinguishing  between  truly  innovative  solutions  and  the 
seller’s  overly  optimistic  cost  estimates. 

Incumbent  sellers  may  enjoy  a  significant  advantage  over 
new  competitors  entering  the  market.  A  new  entrant’s 
optimistic  cost  estimates,  however,  can  win  a  competition 
against  the  accurate  estimates  of  a  lower  cost  incumbent, 
resulting  in  dramatically  higher  final  costs  to  the  buyer 
relative  to  sole-source  procurement  from  the  incumbent. 
When  promoting  competition  against  a  strong  incumbent, 
DoD  acquisition  officials  must  consider  whether  a  new 
entrant  offers  innovation  that  can  realistically  overcome 
the  cost  advantage  enjoyed  by  the  incumbent. 
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Conclusions 

Kendall  (2012),  in  his  memorandum  entitled  “Better  Buying  Power 
2.0,”  reemphasized  competition  to  control  and  reduce  costs.  This  study 
and  a  prior  analysis  of  competition  cast  doubt  on  cost  as  a  rationale  for 
promoting  competition  in  defense  acquisition. 

Harrison  (2012)  concluded  that  maintaining  competition  during  a 
DoD  acquisition  will  likely  increase  the  total  cost  to  the  government.  The 
higher  costs  in  Harrison’s  study  result  from  redundant  development  and 
infrastructure  costs  and  the  inability  of  multiple  competitors  to  benefit 
fully  from  the  cost  savings  of  production  learning  curves. 

This  study  expanded  on  Harrison’s  work  by  evaluating  the  effects 
of  competitor  differences  and  imperfect  knowledge  of  development  and 
production  costs.  While  competition  can  pressure  sellers  to  win  busi¬ 
ness  by  trimming  their  profit  margins,  the  already  low  profit  margins  in 
the  defense  industry  limit  this  cost  improvement  to  just  a  few  percent 
relative  to  the  cost  of  sole-source  procurement. 

Unlike  commercial  markets,  where  sellers  typically  set  prices  after 
completing  product  development  and  initial  production,  sellers  in  the 
defense  acquisition  market  usually  set  their  prices  in  advance  of  devel¬ 
opment  and  production.  The  inaccuracies  and  underestimation  of  these 
costs  in  a  competition  may  appear  to  provide  an  initial  cost  benefit  for  an 
FP  contract,  but  competitive  selection  of  the  lowest  cost  seller  increases 
the  chances  of  unsustainably  low  profits  (or  even  losses)  for  the  seller, 
threatening  program  completion  and  supplier  health.  Bid  inaccuracy 
in  a  competition  for  a  CP  contract  does  not  significantly  affect  the  final 
cost  to  the  government,  but  does  amplify  the  chances  of  cost  overruns 
relative  to  sole-source  procurement,  increasing  planning  challenges  and 
threatening  program  viability. 

Only  when  one  or  more  competitors  offer  innovations  that  truly 
reduce  the  costs  of  development  and  production  does  the  government 
substantially  benefit  from  competition  over  sole-source  procurement 
without  the  adverse  side  effects  of  cost  overruns.  Distinguishing  between 
true  innovation  and  optimistic  cost  estimating,  however,  can  pose  a  chal¬ 
lenge  for  DoD  acquisition  officials. 
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In  defense  markets  with  a  strong  incumbent  that  enjoys  advantages 
of  experience  and  expertise,  forcing  competition  from  less  experienced 
new  entrants  can  produce  costly  final  outcomes.  Incumbents  will  likely 
produce  more  accurate  estimates  reflecting  their  true  cost  advantages, 
while  a  new  entrant  out  of  ignorance  or  competitive  pressure  could 
significantly  underestimate  the  effort  required  and  produce  a  bid  that 
appears  to  beat  the  incumbent.  Unless  the  new  entrant  offers  innovation 
that  overcomes  the  incumbent’s  advantage,  the  cost  of  the  competition 
could  greatly  exceed  sole-source  procurement  from  the  incumbent. 

Certainly,  justification  for  promoting  competition  in  DoD  acqui¬ 
sition  goes  well  beyond  cost  control.  Competition  promotes  fairness 
and  impedes  collusion  by  treating  all  sellers  equally.  It  reassures  the 
citizenry  that  the  government  spends  public  funds  effectively  and  fairly. 
Competition  in  DoD  acquisition  maintains  consistency  with  the  funda¬ 
mentals  of  capitalism  that  drive  the  U.S.  economy,  and  it  can  incentivize 
innovation  in  the  defense  industry  to  provide  improved  capabilities  and 
lower  costs  to  the  DoD. 

These  and  many  other  reasons  can  justify  promoting  competition, 
but  the  analysis  herein  and  by  Harrison  make  cost  savings  from  compe¬ 
tition  an  uncertain  claim.  The  DoD  may  actually  incur  increased  costs 
from  competition,  which— like  a  placebo— creates  a  powerful  perception 
of  cost  control. 
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Recent  legislation,  such  as  the  Weapon  Systems  Acquisi¬ 
tion  Reform  Act  of  2009,  requires  a  renewed  emphasis 
on  understanding  operating  and  support  (O&S)  costs. 
Conventional  wisdom  within  the  acquisition  community 
suggests  a  70:30  cost  ratio  with  respect  to  O&S  and 
acquisition  of  an  average  weapon  system.  Using  37  Air 
Force  and  Navy  programs,  the  authors  estimate  the 
mean  overall  ratio  of  O&S  costs  to  acquisition  costs 
to  be  closer  to  55:45,  although  many  weapon  systems 
displayed  significant  deviation  from  this  55  percent 
average.  Contributing  factors  such  as  life  expectancy 
and  acquisition  strategy  (i.e.,  new  system  or  modifica¬ 
tion)  affect  this  variance.  Their  research  advises  against 
using  a  single  “one-size-fits-all”  O&S/acquisition  cost 
ratio  for  all  major  DoD  weapon  systems. 
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The  Weapon  Systems  Acquisition  Reform  Act  of  2009  and  tight¬ 
ening  Department  of  Defense  (DoD)  budgets  have  brought  increased 
scrutiny  to  the  life-cycle  cost  of  major  weapon  systems  acquisition.  In  a 
significant  paradigm  change,  operating  and  support  (O&S)  costs  are  no 
longer  relegated  to  the  background  for  major  acquisition  decisions.  For 
example,  the  DoD’s  2013  budget  plan  considered  mothballing  the  Block 
30  variant  of  the  Global  Hawk  to  save  money  due  to  O&S  costs,  arguing 
that  the  venerable  U-2  aircraft  could  meet  theater  commanders’  needs 
for  reconnaissance  at  less  cost.  As  a  result,  it  is  imperative  that  we  have 
an  accurate  understanding  of  the  relative  costs  to  operate  and  support 
DoD  weapon  systems. 


FIGURE  1.  NOTIONAL  LIFE-CYCLE  COSTS  FOR  A  DOD  WEAPON 
SYSTEM 


Engineering/ 
Manufacturing 
Development  Phase 


Note.  Figure  1  is  illustrative  versus  quantitative.  Adapted  from  Operating  and  Support 
Cost-Estimating  Guide,  published  by  the  Office  of  the  Secretary  of  Defense,  Cost 
Analysis  Improvement  Group,  1992. 

The  cost  profile  of  atypical  DoD  weapon  system  is  shown  in  Figure 
1  (Office  of  the  Secretary  of  Defense  Cost  Analysis  Improvement  Group 
[OSD  CAIG],  1992).  The  graph  shows  the  four  phases  of  a  program’s  cost 
over  its  lifetime:  research  and  development  (R&D),  procurement,  O&S, 
and  disposal,  with  O&S  considered  the  most  expensive  of  the  four  phases. 
The  conventional  wisdom  of  this  70:30  or  “golden  ratio”  of  O&S  to  acqui¬ 
sition  cost  (assuming  negligible  disposal  cost)  is  that  such  a  pattern  holds 
for  a  majority  of  weapon  systems.  Therefore,  Figure  1  has  permeated 
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the  DoD  literature  and  acquisition  schoolhouse  training  material.  As  a 
result,  many  levels  of  acquisition  leadership  reinforce  the  idea  to  man¬ 
agers  and  analysts  that  a  cost  ratio  exists  among  the  various  stages  of  a 
weapon  system’s  life,  namely  70  percent  for  O&S  and  30  percent  toward 
acquisition  (Carter,  2011). 

Several  studies  by  the  Government  Accountability  Office  (GAO) 
cite  this  70:30  ratio  or  display  charts  to  reflect  this  pattern  (General 
Accounting  Office,  2000a,  2000b;  GAO,  2010, 2012).  This  research  looks 
to  determine  the  origins  and  accuracy  of  this  ratio  using  historical  O&S 
cost  data.  If  actual  O&S  data  do  not  support  this  ratio,  then  the  veracity  of 
this  rule  is  called  into  question  and  might  have  significant  implications 
in  portfolio  analysis  and  affordability  analysis  decisions  that  affect  the 
broader  DoD  budget. 

Terms  and  Definitions 

Throughout  this  article,  we  define  O&S  costs  in  the  same  manner  as 
the  OSD  CAIG  (2007),  now  the  Cost  Assessment  and  Program  Evaluation 
(CAPE)  system’s  O&S  cost: 

Consists  of  sustainment  costs  incurred  from  the  initial  system 
deployment  through  the  end  of  system  operations.  Includes  all 
costs  of  operating,  maintaining,  and  supporting  a  fielded  system. 
Specifically,  this  consists  of  the  costs  (organic  and  contrac¬ 
tor)  of  personnel,  equipment,  supplies,  software,  and  services 
associated  with  operating,  modifying,  maintaining,  supplying, 
training,  and  supporting  a  system  in  the  DoD  inventory,  (p.  2-2) 

For  the  definition  of  life-cycle  cost  (LCC),  we  turn  to  the  Defense 
Acquisition  Guidebook  (DAG).  The  DAG,  published  by  the  Defense 
Acquisition  University  (DAU,  2012),  defines  LCC  as  follows: 

For  a  defense  acquisition  program,  life-cycle  cost  consists  of  R&D 
costs,  investment  costs,  operating  and  support  costs,  and  disposal 
costs  over  the  entire  life  cycle.  These  costs  include  not  only  the 
direct  costs  of  the  acquisition  program,  but  also  indirect  costs 
that  would  be  logically  attributed  to  the  program.  In  this  way,  all 
costs  that  are  logically  attributed  to  the  program  are  included, 
regardless  of  funding  source  or  management  control,  (p.  7) 


Defense  ARJ,  January  2014,  Vol.  21  No.  1: 442-464 


444 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


When  dealing  with  the  life  of  a  weapon  system,  we  discuss  its  ser¬ 
vice  life  and  its  life  expectancy.  According  to  the  DAU  online  glossary, 
the  service  life  describes  the  period  of  time  “from  first  inception  of  the 
weapon  until  final  phase-out”  (DAU,  2012).  Realistically,  some  costs 
incurred  in  the  very  early  stages  of  a  program,  such  as  those  before 
Milestone  A,  may  not  be  fully  captured  due  to  the  immaturity  of  the 
technology  or  divergence  from  some  original  concept.  According  to 
the  1992  and  2007  versions  of  the  OSD  CAIG  (CAPE)  Operating  and 
Support  Cost-Estimating  Guides,  life  expectancy  should  include  the 
phase-in  period,  a  period  of  steady-state  operations,  and  a  phase-out  or 
decommissioning  period  (OSD  CAIG,  1992;  2007).  The  draft  2012  OSD 
CAPE  Operating  and  Support  Cost-Estimating  Guide  wasn’t  as  clear, 
though  it  stated  that  “[t]he  O&S  estimate  should  extend  over  the  full  life 
expectancy  of  the  system,”  alluding  to  the  idea  that  life  expectancy  only 
pertains  to  the  O&S  phase.  As  we  show  in  the  next  section,  these  terms 
appear  to  be  used  interchangeably  even  though  they  are  clearly  defined 
to  be  different  in  scope. 

We  make  one  last  distinction  before  highlighting  various  studies 
that  discussed  acquisition  cost  to  O&S  cost  ratio.  In  performing  financial 
analyses,  analysts  and  researchers  need  to  account  for  inflation  when 
comparing  fiscal  events  that  happened  in  different  time  periods.  The 
Base  Year  (BY),  or  Constant  Year,  describes  past  and  future  costs  as  they 
would  appear  in  a  certain  year  of  reference.  Then  Year  (TY),  or  Current 
Year,  describes  costs  as  they  would  appear  when  costs  are  incurred  or 
when  purchases  are  made,  usually  taking  into  account  the  effects  of 
inflation  over  time.  In  this  research,  we  assume  BY  forms  the  basis  for 
analysis  in  the  literature  reviewed,  unless  specifically  noted. 

Historical  Research 

To  understand  the  origins  of  the  70:30  ratio,  we  conducted  a  lit¬ 
erature  search.  What  was  remarkable  about  this  review  is  how  little 
empirical  research  appears  to  have  been  conducted  on  this  topic,  and 
how  a  recurring,  authoritative  set  of  assertions  continues  to  propogate 
without  independent  evaluation  or  confirmation. 

Two  studies  from  the  1970s  examined  O&S  cost  ratios  with  respect 
to  life-cycle  costing.  Fiorello  (1975)  states  that  the  costs  of  ownership, 
“...in  general  make  up  over  50  percent  of  the  LCC  of  aircraft  weapon 
systems”  (p.  5).  Unfortunately,  Fiorello  provides  no  information  on  the 
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derivation  of  this  percentage.  In  October  1977,  the  Comptroller  General 
of  the  United  States  gave  a  report  to  the  U.S.  Senate  Committee  on 
Appropriations  about  O&S  costs  of  new  systems  compared  to  the  systems 
they  are  replacing  (General  Accounting  Office,  1977).  In  Appendix  IV, 
Part  2,  this  report  shows  the  most  recent  cost  estimate  for  a  fleet  of  800 
F-18s,  and  shows  that  42.2  percent  of  this  fleet’s  LCC  can  be  attributed 
to  O&S  costs.  This  information  was  based  on  an  estimate  that  used  the 
actual  performance  and  logistics  of  the  F-14  as  an  analogy  to  the  F-18, 
and  used  an  estimated  life  span  of  15  years. 


To  understand  the  origins  of  the  70:30  ratio, 
we  conducted  a  literature  search.  What  was 
remarkable  about  this  review  is  how  little  empirical 
research  appears  to  have  been  conducted  on  this 
topic,  and  how  a  recurring,  authoritative  set 
of  assertions  continues  to  propogate  without 
independent  evaluation  or  confirmation. 


In  1981,  the  Comptroller  General  of  the  United  States  delivered 
a  report  to  Congress  on  logistics  planning  for  the  Ml  tank  (General 
Accounting  Office,  1981).  The  report  was  aimed  at  convincing  Congress 
that  more  funding  should  be  spent  on  R&D  and  initial  procurement  to 
reduce  the  O&S  costs,  arguing  “the  costs  of  operating  and  supporting 
a  system,  such  as  the  Ml,  may  be  70  to  90  percent  of  the  system’s  life- 
cycle  cost”  (p.  18).  Like  previous  studies,  the  authors  do  not  elaborate  or 
indicate  the  source  of  this  ratio  information. 

With  the  release  of  the  Operating  and  Support  Cost-Estimating 
Guide  in  1992,  OSD  CAIG  gave  more  official  guidance  regarding  O&S 
cost  estimates  (OSD  CAIG,  1992).  This  guide  does  not  designate  any 
particular  ratio  of  O&S  costs  to  acquisition  costs,  but  does  portray  the 
customary  program  cost  pattern  during  the  various  acquisition  phases. 
Figure  1  originated  within  this  1992  guide,  and  many  training  materials 
have  reproduced  or  mimicked  this  graph.  Since  1992,  the  OSD  CAIG  has 
issued  one  other  Operating  and  Support  Cost-Estimating  Guide  (2007) 
and  had  intended  to  officially  release  another  in  2012.  We  reviewed 
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the  2012  draft  version,  but  the  OSD  CAPE  Operating  and  Support 
Cost-Estimating  Guide  delayed  final  publication  due  to  the  impending 
release  ofthe  revised  DoD  5000.4-M-l,  Cost  and  Software  Data  Reporting 
(CSDR)  Manual  (2007),  to  incorporate  any  policy  changes  therein.  Figure 
2,  which  first  appears  in  the  2007  and  also  the  2012  draft  OSD  CAPE  O&S 
Guide  illustrates  the  slight  change  to  Figure  1  from  1992.  Neither  of  these 
versions  of  the  guide  includes  any  further  information  on  cost  ratios. 


FIGURE  2.  ILLUSTRATIVE  SYSTEM  LIFE  CYCLE 


< — ►  Engineering  and  Manufacturing  <_ 

Materiel  Solution  Development  Phase  Disposal  Phase 

Analysis  and  Technology  < - *. 

Development  Phases  Production  and  Deployment  Phase 

Sustainment  Phase 

PROGRAM  LIFE  CYCLE  (ILLUSTRATIVE) 

Note.  This  figure  depicts  the  ratio  as  nominal.  Adapted  from  Operating  and  Support 
Cost-Estimating  Guide,  published  by  the  Office  of  the  Secretary  of  Defense,  Cost 
Analysis  Improvement  Group,  1992. 

In  1997,  the  Defense  Systems  Management  College  (DSMC)  pub¬ 
lished  its  Acquisition  Logistics  Guide,  in  which  it  illustrates  “the 
dominant  role  that  logistics  plays  in  system  life-cycle  cost”  (DSMC, 
1997),  as  portrayed  in  Figure  3.  This  is  the  first  time  a  ratio  with  this 
level  of  specificity  is  provided  (72  percent  of  life-cycle  costs  attributed  to 
O&S).  Unfortunately,  the  guide  provides  no  details  on  how  the  percent¬ 
ages  were  obtained  or  derived.  Figure  3  is  replicated  in  four  other  sources: 
(a)  a  2000  General  Accounting  Office  report  titled  Air  Force  Operating 
and  Support  Cost  Reductions  Need  Higher  Priority  (General  Accounting 
Office,  2000a);  (b)  a  2003  General  Accounting  Office  report  on  reduc¬ 
ing  Total  Ownership  Costs  through  setting  requirements  (General 
Accounting  Office,  2003);  (c)  the  American  Institute  of  Aeronautics  and 
Astronautics’  Management  of  Defense  Acquisition  Projects  (Rendon, 
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Snider,  &  Allen,  2008);  and  (d)  an  acquisition  research  paper  published 
by  the  Naval  Postgraduate  School  entitled,  Total  Ownership  Cost— Tools 
and  Discipline  (Naegle  &  Boudreau,  2011). 


FIGURE  3.  NOMINAL  LIFE-CYCLE  COSTS 
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Note.  Adapted  from  Figure  13-1  of  Acquistion  Logistics  Guide  (3rd  ed.),  Defense  Systems 
Management  College,  1997,  p.  13-6. 


In  1999,  the  Institute  for  Defense  Analyses  (IDA)  produced  a  seem¬ 
ingly  influential  document  that  covered  a  presentation  by  a  panel  of 
representatives  from  the  OSD,  Naval  Center  for  Cost  Analysis,  Air  Force 
Cost  Analysis  Agency,  and  the  U.S.  Army  Cost  and  Economic  Analysis 
Center  during  the  32nd  Annual  DoD  Cost  Analysis  Symposium  (IDA, 
1999).  In  this  document,  weapon  system  types  are  split  out  and  presented 
in  terms  of  their  R&D,  procurement,  and  O&S  costs,  where  the  informa¬ 
tion  is  available.  Table  1  summarizes  the  information  presented,  which 
is  cited  in  the  Life-Cycle  Cost  article  from  the  DAU’s  ACQuipedia  Web 
site  (Life-Cycle  Cost,  2008). 


For  most  system  types,  the  percentages  reflect  what  was  considered 
at  the  time  to  be  “typical”  percentages  of  life-cycle  costs.  The  excep¬ 
tions  were  in  the  Rotary  Wing  Aircraft  category,  where  the  percentages 
came  from  the  Comanche  estimate  in  the  1997  Selected  Acquisition 
Report  (SAR),  and  the  Missiles  and  Surface  Vehicles  categories,  which 
did  not  specifically  state  what  the  percentages  represent.  However,  we 
assumed  them  to  be  “typical”  since  no  other  discussion  led  us  to  believe 
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otherwise.  The  only  two  categories  that  come  close  to,  or  meet  exactly, 
the  70:30  ratio  are  the  Ships  and  Automated  Information  Systems  (AIS) 
categories.  The  data  from  Table  1  appear  to  be  the  source  for  the  GAO 
Cost-Estimating  and  Assessment  Guide  (GAO,  2009),  and  DAU’s  course 
material  on  BCF 106,  Introduction  to  Cost  Analysis  (DAU,  2009). 


TABLE  1.  COST  RATIOS  BY  WEAPON  SYSTEM  TYPE 


System  Type 

R&D 

Investment 

O&S/Disposal 

Space 

18% 

66% 

16% 

Fixed-Wing  Aircraft 

20% 

39% 

41% 

Rotary-Wing  Aircraft 

15% 

52% 

33% 

Missiles 

27% 

33% 

39% 

Electronics 

22% 

43% 

35% 

Ships” 

1% 

31% 

68% 

Surface  Vehicles 

9% 

37% 

54% 

AISb 

30% 

70% 

Note.  Data  represent  point  estimates,  without  accompanying  statistical  data.  No 
further  information  was  obtainable.  Adapted  from  Status  of  DoD's  Capability  to 
Estimate  the  Costs  of  Weapon  Systems:  1999  Update,  published  by  the  Institute  for 
Defense  Analyses,  1999. 

“Most  ship  design  costs  are  included  in  production  cost  of  lead  ship  of  a  class.  "Available 
data  preclude  split  of  pre-O&S  costs  into  R&D  and  Investment  categories. 

For  the  past  15  years  or  so,  the  GAO  consistently  cites  or  mentions 
this  70:30  ratio  of  acquisition  cost  to  O&S  cost  (General  Accounting 
Office,  2000a).  Specifically: 

...operating  and  support  costs  include  those  for  fuel,  repair  parts, 
maintenance,  and  contract  services,  as  well  as  the  costs  of  all 
civilian  and  military  personnel  associated  with  a  weapon  sys¬ 
tem.  History  indicates  that  these  costs  can  account  for  about  70 
percent  of  a  system’s  total  life-cycle  costs,  (p.  3) 

With  respect  to  the  Army  (General  Accounting  Office,  2000b): 

While  some  attention  has  been  given  to  the  cost  of  operating  and 
supporting  a  weapon  system  after  it  is  fielded,  responsibility 
for  these  functions  after  systems  are  fielded  generally  shifts  to 
other  Army  agencies  such  as  maintenance  depots,  software  sup¬ 
port  facilities,  and  operating  bases.  DoD  has  long  identified  this 
division  of  responsibility  as  a  key  cause  of  higher  weapon  system 
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operating  and  support  costs,  which  are  generally  estimated  to 
account  for  about  60  to  70  percent  of  a  system’s  total  life-cycle 
costs,  (p.  7) 

GAO  is  not  the  only  recent  source  to  focus  on  this  particular  ratio 
value.  In  an  article  in  the  Defense  AT&L  magazine  on  designing  sys¬ 
tems  for  supportability,  Dallosta  and  Simcik  (2012)  state  that,  “...total 
ownership  costs  incurred  during  the  operations  and  support  phase  may 
constitute  65  percent  to  80  percent  of  total  life-cycle  cost.”  Figure  4 
accompanies  this  quote  within  their  article,  but  once  again,  no  informa¬ 
tion  is  provided  on  how  that  figure  was  derived  or  percentages  allocated. 


FIGURE  4.  NOMINAL  LIFE-CYCLE  COST  DISTRIBUTION 

< -  LIFE-CYCLE  COST  - 


Operating  and  Support 


Systems  Acquisition 


i 


1 


- . V  60%-80%  \ 

♦  ♦  ♦♦  ♦ 


y  20%-40%  ♦*  V 


30+ YEARS 


Note.  Adapted  from  Designing  for  Supportability:  Driving  Reliability,  Availability,  and 
Maintainability  In  While  Driving  Costs  Out,  by  P.  M.  Dallosta  and  T.  A.  &  Simcik,  2012,  pp. 
34-38. 


The  history  of  O&S  cost  ratios,  as  presented  in  the  reports  and  stud¬ 
ies  discussed  in  this  article  and  summarized  in  Table  2,  often  show  the 
plausibility  of  70  percent  of  a  total  weapon  system’s  LCC  representing 
O&S  costs,  especially  in  the  more  recent  reports.  In  addition,  relatively 
few  of  the  O&S  statistics  cited  in  the  Table  2  literature  review  appear  to 
be  grounded  in  historical  O&S  data.  Instead,  we  find  they  are  based  on 
estimates  of  how  long  a  weapon  system  will  last  and  how  costly  it  is  to 
repair,  replace,  sustain,  maintain,  or  operate.  By  extracting  actual  O&S 
cost  data  and  accounting  for  the  increased  length  of  current  weapon 
systems,  serious  researchers  can  readily  determine  what  this  true  per¬ 
centage  should  be. 
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TABLE  2.  SUMMARY  OF  FINDINGS  FROM  LITERATURE  REVIEW 


Source 

O&S  Portion  of  LCC 

Fiorello,  M„  Getting  "Real"  Data  for 

Life-Cycle  Costing,  1975 

50% 

General  Accounting  Office,  O&S  Costs  of 

New  Weapon  Systems  Compared  with  Their 
Predecessors,  1977 

42.2% 

General  Accounting  Office,  Logistics 

Planning  for  the  Ml  Tank,  1981 

70-90% 

OSD  CAIG,  O&S  Cost-Estimating  Guide, 

1992 

78%,  84% 

DSMC,  Acquisition  Logistics  Guide,  1997 

60-80%,  72% 

IDA,  Status  of  DoD's  Capability  to  Estimate 
the  Costs  of  Weapon  Systems:  1999  Update, 

1999 

Varies  by  Type 

General  Accounting  Office,  Higher  Priority 
Needed  for  Army  O&S  Cost  Reduction 

Efforts,  2000 

60-70% 

DoD,  Weapon  Systems  Acquisition  Reform 

Act  of  2009 

60-75% 

General  Accounting  Office,  Cost  Estimating 

and  Assessment  Guide:  Best  Practices  for 

Developing  and  Managing  Capital  Program 
Costs,  2009 

Varies  by  Type 

General  Accounting  Office,  Littoral  Combat 
Ship:  Actions  Needed  to  Improve  Operating 
Cost  Estimates  and  Mitigate  Risks  in 
Implementing  New  Concepts,  2010 

70% 

General  Accounting  Office,  Improvements 
Needed  to  Enhance  Oversight  of  Estimated 
Long-Term  Costs  for  Operating  and 
Supporting  Major  Weapon  Systems,  2012 

70% 

Dallosta  &  Simcik,  Designing  for 
Supportability:  Driving  Reliability, 

Availability,  and  Maintainability  In  While 
Driving  Costs  Out,  2012 

65-80% 
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Analysis  and  Results 

To  collect  actual  O&S  expenditures,  we  utilized  the  Naval  Visibility 
and  Management  of  Operating  and  Support  Costs  (VAMOSC)  system  for 
the  Navy,  and  the  Air  Force  Total  Operations  Cost  (AFTOC)  system  for 
the  Air  Force.  We  excluded  Army  programs  from  the  analysis  because 
the  Army’s  Operating  and  Support  Management  Information  System  is 
currently  unable  to  allocate  major  cost  elements  (including  personnel 
and  fuel)  to  individual  systems.  Acquisition  costs  were  collected  from 
SARs  in  the  Defense  Acquisition  Management  Information  Retrieval 
system.  The  analysis  was  limited  to  Acquisition  Category  I  (ACAT  I) 
programs.  [Note.  These  are  programs  that  exceed  $365  million  (BY  2000) 
in  Research,  Development,  Test,  &  Evaluation  (RDT&E)  funding  or  $2.19 
billion  (BY  2000)  in  Procurement  funding,  or  have  been  designated  by 
Congress  or  the  DoD  as  an  ACAT  I  program  due  to  high  visibility  or  inter¬ 
est.]  Generally,  costs  associated  with  necessary  additions  or  changes 
for  each  system  were  included  in  the  SARs,  mostly  under  the  Military 
Construction  appropriation.  Infrastructure  costs  were  not  necessary  for 
many  systems  since  some  new  and  many  modification  programs  do  not 
require  new  facilities  or  structures. 

The  data  were  screened  using  inclusion  criteria  for  the  research  data¬ 
base.  Each  program  had  to  have  fielded  operational  units  and  have  a  stable 
period  of  O&S  costs.  This  stability  provided  some  assurance  that  the 
program  was  past  the  initial  ramp  -up  in  fielding  and  was  able  to  produce 
a  realistic  estimate  of  recurring  annual  costs.  Therefore,  each  program 
needed  to  have  produced  at  least  10  percent  of  the  planned  procurement 
quantities.  Early  in  production,  contractors  may  run  into  difficulties  that 
could  change  the  production  schedule  or  increase  costs  due  to  factors 
unknown  when  production  commences.  Until  these  issues  are  resolved, 
the  acquisition  is  likely  to  have  a  greater  risk  of  increasing  significantly. 

The  final  database  consisted  of  37  programs  with  operational  data 
from  1989  through  2010.  Table  3  lists  all  the  programs  analyzed,  organized 
by  the  lead  Service  component.  We  grouped  these  programs  into  eight 
different  categories:  Missiles,  Cargo/Tanker  Aircraft,  Fighter  Aircraft, 
Helicopters,  Ships,  Electronic  Equipment,  Unmanned  Aerial  Vehicles, 
and  Tilt-Rotor  Aircraft.  We  determined  these  categories  by  similarities  or 
unique  capabilities  from  other  weapon  systems;  for  example,  the  tilt-rotor 
aircraft,  which  is  a  combination  of  helicopter  and  cargo  aircraft. 
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TABLE  3.  FINAL  DATABASE-LIST  OF  PROGRAMS  ANALYZED 


Ships 

Service 

AOE  6 

Navy 

CVN  68 

(By  1974/1975) 

Navy 

CVN  68  (By  1976) 

Navy 

DDG  51 

Navy 

LHD  1 

Navy 

LPD  17 

Navy 

MHC  51 

Navy 

SSGN 

Navy 

SSN  21 

Navy 

SSN  774 

Navy 

STRATEGIC 

SEALIFT 

Navy 

T-AKE 

Navy 

T-AO  187 

Navy 

Fighters 

Service 

F-16  C/D 

Air  Force 

F-22  (BY  2005) 

Air  Force 

JPATS  (BY  2002) 

Air  Force/Navy 

AV-8B  REMAN 

Navy 

EA-18G 

Navy 

F/A-18  E/F 

Navy 

F-14D 

Navy 

T-45TS  (BY  1995) 

Navy 

Cargo/Tanker 

Service 

C-130J 

Air  Force 

C-17A  (BY  1996) 

Air  Force 

E-2C 

Navy 

JSTARS  (BY  1998) 

Air  Force/Army 

KC-135R 

Air  Force 

Missiles 

Service 

AMRAAM 

Air  Force/Navy 

JASSM 

Air  Force/Navy 

AIM-9X 

Navy/Air  Force 

JSOW  (AGM-154) 

Navy/Air  Force 

Helicopters 

Service 

C/MH-53E 

Navy 

MH-60R 

Navy 

(BY  2006) 

MH-60S 

Navy 

UVA 

Service 

GLOBAL  HAWK 

Air  Force 

PREDATOR 

Air  Force 

Electronic 

Equipment 

Service 

NESP 

Navy 

Tilt-Rotor  Service 

V-22  (BY  2005)  Navy 
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From  the  information  in  the  VAMOSC  and  AFTOC  systems,  we  cal¬ 
culated  an  actual  Annual  Unitized  O&S  Cost  (AUC)  per  program.  This 
metric  describes  the  cost  to  operate  and  sustain  one  unit  (individual 
plane,  ship,  etc.)  per  year.  Generally  speaking,  the  AUC  is  calculated  by 
dividing  the  total  annual  O&S  cost  for  a  system  by  the  number  of  units 
operational  in  the  year.  [Note.  For  a  more  complete  description  of  the 
AUC  methodology,  see  Ryan,  Jacques,  Colombi,  &  Schubert  (2012).]  We 
approximated  the  total  O&S  cost  of  a  particular  program  by  multiply¬ 
ing  the  AUC  by  the  number  of  units  procured  by  the  life  expectancy  of 
the  system.  It  is  important  to  note  that  some  costs  that  can  be  logically 
attributable  to  programs,  such  as  the  maintenance  of  simulators  and 
training  devices,  may  or  may  not  be  included  properly  in  the  VAMOSC 
and  AFTOC  systems.  This  uncertainty  has  the  potential  for  understat¬ 
ing  O&S  costs. 


TABLE  4.  SUMMARY  OF  DIFFERENCES  IN  LIFE  EXPECTANCY 
FROM  VARIOUS  VERSIONS  OF  THE  OSD  CAIG  O&S  COST¬ 
ESTIMATING  GUIDE 


1992 

2007 

2012  (draft) 

Cargo 

25 

25 

30-40 

Bomber 

25 

25 

30-40 

Tanker 

25 

25 

30-40 

Fighter 

20 

20 

20-30 

Flelicopter 

20 

20 

20-30 

Small  Missiles 

15 

15 

10-20 

Large  Missiles 

20 

15 

10-20 

Electronic  Equipment 

10 

10 

10-30 

Ships 

20-40 

20-40 

20-40 

Ground  Combat  Vehicles 

20 

20 

20 

Unmanned  Aerial  Vehicles 

N/A 

N/A 

15-25 

Table  4  shows  how  the  life  expectancies  have  changed  over  the  years. 
As  illustrated,  life  expectancy  has  increased  for  most  systems,  most 
notably  for  the  Cargo/Bomber/Tanker  and  Electronic  Equipment  cat¬ 
egories.  Since  one  of  the  unknowns  in  this  analysis  is  the  expected  life  of 
each  program,  we  used  the  draft  2012  OSD  CAPE  Operating  and  Support 
Cost-Estimating  Guide,  coupled  with  program-specific  information 


Defense  ARJ,  January  2014,  Vol.  21  No.  1: 442-464 


454 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


found  in  SARs,  to  develop  specific  platform  service  life  ranges.  Table  5 
shows  these  life  expectancy  ranges.  From  these  estimates,  we  chose  the 
highest  and  lowest  expectancies  to  use  as  an  upper  and  lower  bound. 


TABLE  5.  LIFE  EXPECTANCIES  FOR  VARIOUS  WEAPON  SYSTEMS 


High 

Low 

High 

Low 

(yrs) 

(Yrs) 

(yrs) 

(Yrs) 

Ships 

Cargo/Tanker 

AOE  6 

40 

20 

C-130J 

50 

25 

CVN  68 

50 

20 

C-17A  (BY  1996) 

40 

25 

(BY  1974/1975) 

E-2C 

40 

20 

CVN  68  (BY  1976) 

50 

20 

JSTARS  (BY  1998) 

40 

25 

DDG  51 

40 

20 

KC-135R 

40 

25 

LHD  1 

40 

20 

LPD  17 

40 

20 

Missiles 

MHC  51 

40 

20 

AMRAAM 

40 

10 

SSGN 

40 

20 

JASSM 

20 

10 

SSN  21 

40 

20 

AIM-9X 

33 

10 

SSN  774 

40 

20 

JSOW  (AGM-154) 

30 

10 

STRATEGIC 

40 

20 

Helicopters 

SEALIFT 

C/MH-53E 

30 

20 

T-AKE 

40 

20 

MH-60R  (BY  2006) 

30 

20 

T-AO  187 

40 

20 

MH-60S 

35 

20 

Fighters 

UVA 

F-16  C/D 

30 

20 

GLOBAL  HAWK 

34 

15 

F-22  (BY  2005) 

30 

20 

PREDATOR 

25 

15 

JPATS  (BY  2002) 

30 

20 

Electronic  Equipment 

AV-8B  REMAN 

30 

20 

NESP 

30 

10 

EA-18G 

30 

20 

Tilt-Rotor 

F/A-18  E/F 

30 

20 

V-22  (BY  2005) 

43 

30 

F-14D 

30 

20 

T-45TS  (BY  1995) 

30 

20 
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Before  performing  any  calculations,  we  standardized  the  annual  cost 
data  to  Fiscal  Year  (FY)  2010  using  OSD  inflation  indices.  Once  normal¬ 
ized  to  FY  2010,  the  costs  per  year  were  de-escalated  back  to  the  base 
year  of  the  program.  For  certain  programs  that  reported  more  than  one 
baseline  year  due  to  changes  or  milestones  in  the  program,  for  example, 
the  V-22  Osprey,  we  used  the  most  recent  SAR  report.  Multiplying  the 
actual  AUC  by  the  highest  (or  lowest)  life  expectancy  for  a  program  and 
by  the  number  of  units  to  be  procured  (as  given  by  the  last  or  most  recent 
SAR)  resulted  in  our  estimate  of  O&S  costs.  To  calculate  the  ratio  of  O&S 
to  LCC,  we  divided  the  O&S  cost  estimate  by  the  total  of  the  O&S  cost 
estimate  and  the  acquisition  actual  cost.  Table  6  provides  the  summary 
statistics  for  all  weapon  systems  studied. 


TABLE  6.  SUMMARY  STATISTICS  FOR  ALL  PROGRAMS 


Mean 

Median 

Standard  Deviation 

High 

55.92% 

62.57% 

22.74% 

Low 

43.85% 

48.33% 

21.96% 

Average 

49.88% 

54.09% 

23.02% 

For  all  programs  as  a  whole,  we  estimate  an  approximate  range  of 
44-56  percent  (mean)  or  48-63  percent  (median)  for  the  proportion  of 
life-cycle  costs  attributable  to  O&S.  The  “high”  end  of  the  range  (using 
the  upper  estimate  of  life  expectancy)  went  from  4.91  percent  (Joint 
Standoff  Weapon,  or  JSOW)  to  88.79  percent  (KC-135R),  with  a  standard 
deviation  of  22.48  percent.  The  “low”  end  (using  the  lower  estimate  of 
life  expectancy)  started  at  1.69  percent  (JSOW)  and  went  through  83.19 
percent  (KC-135R),  with  a  standard  deviation  of  21.56  percent.  The  large 
standard  deviation  associated  with  this  overall  range  highlights  vast 
differences  among  the  weapon  systems  in  terms  of  O&S  costs  and  empha¬ 
sizes  the  need  to  further  reduce  the  set  of  programs  into  different  types. 

In  Table  7,  we  segregate  the  weapon  systems  into  eight  categories, 
and  then  for  the  Ship  and  Cargo/Tanker  groups  we  performed  two  addi¬ 
tional  analyses.  The  Ships  category  includes  13  ships,  three  of  which 
consisted  of  submarines.  Of  these  three,  two  fell  outside  two  standard 
deviations  from  the  mean— the  SSN  21  and  the  SSN  774.  The  SSN  21  O&S 
proportion  was  estimated  to  fall  within  11.65  percent  and  20.87  percent, 
and  the  SSN  774  was  estimated  to  fall  between  12.65  percent  and  22.46 
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percent.  Because  these  relatively  low  O&S  percentages  affected  the  mean 
value  for  the  whole  group,  we  reran  the  Ship  group  without  the  subma¬ 
rines  and  presented  that  information  as  well. 


TABLE  7.  SUMMARY  OF  O&S  COST  PERCENTAGES  BY  TYPE  OF 
SYSTEM 


Platform 

Mean 

Median 

Standard  Deviation 

Ships 

48.21% 

51.12% 

18.14% 

Ships  — 

53.26% 

53.12% 

13.13% 

No  Submarines 

Fighter  Aircraft 

52.99% 

51.46% 

15.65% 

Cargo/Tanker  Aircraft 

65.15% 

61.73% 

13.98% 

Cargo/Tanker  — 

59.94% 

59.55% 

9.68% 

No  KC-135R 

Missiles 

8.35% 

6.56% 

7.51% 

Helicopters 

70.73% 

70.13% 

5.70% 

Unmanned  Aerial 

71.56% 

71.56% 

9.39% 

Vehicles 

Electronic  Equipment 

15.53% 

15.53% 

9.60% 

Tilt  Rotor  Aircraft 

65.03% 

65.03% 

5.77% 

For  the  Cargo/Tanker  group,  five  airframes  were  included  in  this  cat¬ 
egory— four  Air  Force  and  one  Navy.  One  program  stood  out  as  anomalous 
in  this  group— the  KC-135R.  Overall,  the  ratios  for  this  category  were 
59.19  percent-71.11  percent  (mean);  and  54.20  percent-70.30  percent 
(median).  The  KC-135R  range  was  83.19-88.79.  Although  the  upper  esti¬ 
mate  for  the  KC-135  fell  within  two  standard  deviations  of  the  mean  and 
median,  the  estimate  for  the  low  end  exceeded  two  standard  deviations 
above  both  measures.  As  with  the  Ship  category,  we  removed  this  outlier 
from  the  group  and  reran  the  analysis.  Lastly,  although  we  did  determine 
a  range  of  the  expected  O&S  proportion  of  LCC  for  electronic  systems,  we 
cannot  in  good  faith  determine  this  range  to  be  representative  for  other 
electronic  systems  given  we  only  had  one  data  point  in  this  category. 

Since  the  work  performed  on  this  database  seemed  to  show  a  possible 
connection  between  high  O&S  proportions  and  variant/modification 
programs,  an  additional  analysis  was  performed  on  all  newly  developed 
systems.  This  is  presumably  due  to  the  fact  that  the  initial  acquisition 
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cost  of  a  plane,  for  instance,  is  included  in  the  “new”  estimate,  but  not 
included  in  the  “modification”  estimate  since  the  aircraft  has  already 
been  purchased.  The  resulting  list  included  22  systems.  The  ranges 
for  O&S  proportions  for  this  group  were  35.09  percent-47.00  percent 
(mean);  and  36.97  percent-53.98  percent  (median).  The  decreases  in 
proportions  from  the  larger  group  of  systems,  including  variant  and 
modification  programs,  seem  to  lend  some  credence  to  the  notion  that 
new  systems  will  have  more  life-cycle  costs  devoted  to  acquisition  than 
to  sustainment.  Table  8  summarizes  the  results. 


TABLE  8.  SUMMARY  STATISTICS  FOR  NEW  PROGRAMS 


Mean 

Median 

Standard  Deviation 

High 

47.00% 

53.98% 

23.60% 

Low 

35.09% 

36.97% 

21.27% 

Average 

41.04% 

45.84% 

22.99% 
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Discussion  and  Conclusions 

To  make  informed  decisions  regarding  the  maintenance  and  lifetime 
cost  of  our  nation’s  weapon  systems,  leaders  and  portfolio  managers  need 
to  have  the  right  information  at  the  right  time.  The  DoD  has  accumulated 
valuable  information  about  its  sustainment  costs  through  systems  like 
AFTOC  and  VAMOSC.  By  tapping  into  this  historical  information, 
analysts  and  decision  makers  can  better  understand  what  portion  of  a 
weapon  system’s  LCC  can  be  attributed  to  acquiring  the  weapon  system 
and  what  portion  can  be  attributed  to  operating  and  supporting  it. 

The  notion  of  O&S  costs  being  70  percent  of  LCC  has  been  circulating 
around  the  DoD  acquisition  community  for  more  than  35  years,  and  has 
repeatedly  been  emphasized  in  several  recent  GAO  reports.  The  origin 
of  this  70:30  ratio  comes  from  an  amalgamation  of  estimates  of  the  O&S 
weapon  systems’  costs  given  by  program  offices  or  other  official  sources, 
such  as  SARs.  However,  by  analyzing  the  actual  sustainment  costs  in 
VAMOSC  and  AFTOC,  the  70  percent  O&S  to  30  percent  Acquisition 
cost  ratio  for  a  “typical”  DoD  weapon  system  appears  not  to  be  valid.  Our 
data  suggest  that  O&S  costs  are  quite  varied,  with  a  mean  of  55  percent. 

Not  only  does  the  conventional  wisdom  regarding  this  fundamental 
LCC  ratio  appeartobe  incorrect,  but  the  tendency  to  reduce  the  life-cycle 
costs  of  all  DoD  weapon  systems  down  to  a  single  ratio  with  respect  to 
acquisition  cost  is  impractical  and  imprudent.  Although  the  average 
percentage  of  O&S  costs  observed  fell  around  50-55  percent  of  LCC,  we 
noticed  significant  deviations  from  this  percentage.  Not  only  did  indi¬ 
vidual  weapon  system’s  ratios  vary  from  this  percentage,  but  also  entire 
categories  of  systems.  Both  of  these  observations  suggest  a  peanut  butter 
spread  of  one  ratio  of  acquisition  to  sustainment  is  too  simplistic.  The 
differences  within  certain  categories  or  subcategories,  such  as  Ships 
and  Submarines,  illustrate  the  need  to  further  distill  these  groups  into 
more  meaningful  and  homogeneous  types  of  systems  before  assigning  a 
typical  O&S/Acquisition  cost  ratio. 

Another  interesting  item  to  come  out  of  this  research  was  the  vari¬ 
able  nature  of  life  expectancies  itself.  As  shown  earlier  in  Table  4,  many 
weapon  systems  categories  have  experienced  an  increase  in  their  recom¬ 
mended  life  expectancies  over  the  past  two  decades.  Fighter  platforms 
expected  to  be  operational  for  15  years,  e.g.,  the  F-15,  are  still  around 
almost  30  years  later.  A  look  at  the  actual  useful  lives  as  well  as  the 
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expected  lives  of  our  weapon  systems  has  shown  that  not  only  are  we 
capable  of  sustaining  our  weapon  systems  far  beyond  their  intended 
lives,  but  we  are  able  to  extend  the  capabilities  of  existing  naval  vessels 
and  airframes  through  modification.  This  can  have  a  profound  impact 
on  the  costs  to  sustain  these  systems  for  a  longer  duration.  As  these 
systems  continue  to  age,  additional  research  should  be  conducted  to 
monitor  actual  O&S  costs. 

By  illustrating  the  variability  of  life-cycle  proportions  among 
weapon  systems  categories,  we  have  shown  a  more  realistic  picture 
of  what  program  analysts  and  portfolio  managers  can  expect  in  terms 
of  sustainment  costs.  Although  beyond  the  scope  of  our  work,  perhaps 
future  studies  can  drill  down  to  speculate  or  reason  why  different  sorts 
of  systems  appear  to  have  such  different  cost  ratios.  This  research  has 
begun  to  open  a  window  into  the  real  effects  of  acquisition  strategy  on 
life-cycle  costs.  In  the  face  of  looming  budget  cuts  over  the  next  decade, 
leaders  across  the  DoD  and  Congress  are  struggling  to  make  tough  deci¬ 
sions  regarding  our  nation’s  arsenal.  Only  with  a  full  understanding  of 
how  our  acquisition  decisions  affect  our  long-term  sustainment  costs  can 
we  make  the  right  decisions  on  what  capabilities  are  needed,  how  we  will 
acquire  those  capabilities,  and  how  we  will  maintain  those  capabilities. 
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Every  year,  the  Department  of  Defense  (DoD)  upgrades 
its  information  technology  systems,  allows  new  appli¬ 
cations  to  connect  to  DoD  information  networks,  and 
reconfigures  the  enterprise  to  gain  efficiencies.  While 
these  actions  better  support  the  warfighter  and  satisfy 
national  security  interests,  they  introduce  new  system 
vulnerabilities  waiting  to  be  exploited.  Often,  these 
vulnerabilities  are  discovered  only  after  the  system  has 
already  deployed,  where  costs  to  fix  are  much  larger. 
This  article  recommends  the  DoD  adopt  an  economic 
strategy  called  the  vulnerability  market,  or  the  market 
for  zero-day  exploits,  to  enhance  system  Information 
Assurance.  Through  the  mutual  cooperation  between 
industry  and  the  military  in  securing  information,  the 
DoD  optimizes  security  investments,  secures  critical 
information,  and  provides  an  effective  and  resilient 
warfighting  capability. 
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To  save  money,  increase  automation,  and  facilitate  information 
sharing,  the  Department  of  Defense  (DoD)  is  increasingly  acquiring 
new  information  system(s),  or  IS.  These  new  systems  are  more  complex, 
interconnected,  and  interdependent  than  older  systems  in  the  DoD 
inventory.  With  these  new  capabilities  comes  a  negative  externality; 
the  more  complex  a  system  is,  the  more  difficult  it  is  to  secure.  Faced 
with  this  reality,  the  United  States  is  making  a  significant  investment  in 
cyber  security.  In  the  years  between  2004  and  2009,  the  annual  federal 
cyber  security  investment  grew  from  $4.2  billion  to  $7.3  billion  (a  58 
percent  increase).  The  augmented  investment  in  cyber  security  focuses 
on  establishing  a  front-line  defense  to  prevent  intrusions,  integrating 
intelligence  into  cyber  security,  and  shaping  the  future  environment  by 
enhancing  research,  development,  and  education.  One  gaping  hole  in  this 
strategy  is  a  focus  on  acquiring  systems  that  are  secure  by  design.  This 
article  is  an  analysis  of  that  gap  and  investigates  whether  the  integra¬ 
tion  of  a  vulnerability  market  (VM),  or  the  market  for  zero-day  exploits, 
increases  overall  DoD  cyber  security  and  lowers  the  total  cost  of  owner¬ 
ship  for  acquired  systems. 

The  Prevalence  of  Vulnerabilities 

Historically  in  the  DoD,  as  budgets  get  tighter,  IS  aggregate.  This  phe¬ 
nomenon  occurs  primarily  to  offset  the  expense  of  maintaining  a  large 
workforce  by  automating  much  of  the  work  accomplished  by  individuals. 
These  systems  also  aggregate  because  of  technological  advances  that 
reduce  their  physical  footprint  and  required  operations  and  maintenance 
(e.g.,  virtualization).  As  a  consequence  of  aggregation,  an  increase  in  the 
number  of  automated  processes  drives  an  increase  in  the  quantity  and 
complexity  of  IS.  Unfortunately,  as  the  number,  complexity,  and  size  of 
systems  increase,  the  prevalence  of  flaws  also  increases. 

A  common  measure  of  the  complexity  of  a  system  is  calculated  by 
enumerating  the  amount  of  software  lines  of  code  (SLOC).  In  2010, 
a  RAND  study  noted  large  code  bases  typically  indicate  a  rate  of  one 
defect  for  every  thousand  lines  of  code  (Landree,  2010).  By  applying  this 
defect  rate  to  two  widely  utilized  operating  systems— Windows  Vista 
and  Debian  Linux— there  would  be  approximately  50,000  defects  in  the 
Microsoft  Windows  Vista  Operating  System,  and  200,000  defects  in 
Debian  Linux  (Marchenko  &  Abrahamsson,  2007).  Applying  this  defect 
rate  to  the  Navy  DD(X)’s  10  million  SLOC,  there  may  be  as  many  as 


467 


Defense ARJ,  January  2014,  Vol.  21  No.  1: 466-484 


Strengthening  DoD  Cyber  Security  with  the  Vulnerability  Market 


10,000  defects.  While  only  a  fraction  of  these  defects  would  allow  access 
to  the  IS  and  lead  to  unauthorized  system  control,  an  entirely  defect-free 
IS  is  realistically  impossible  to  achieve. 

DoD’s  Information  Security  Efforts 

In  response  to  the  enormity  and  potential  consequences  of  a 
state-sponsored  or  independent  hacker  exploiting  critical  system  vul¬ 
nerabilities,  the  DoD  relies  on  a  concept  called  “Defense-in-Depth.” 
Defense-in-Depth  is  the  DoD  approach  to  distributing  system-wide 
exploitation  risk  across  multiple  levels  of  information  security.  The  levels 
integrated  in  this  shared-risk  environment,  according  to  Department 
of  Defense  Directive  (DoDD)  8500.01E,  are:  “people,  technology,  and 
operations;  the  layering  of  I A  [information  assurance]  solutions  within 
and  among  IT  [information  technology]  assets;  and,  the  selection  of  IA 
solutions”  (DoD,  2002).  Stated  simply,  by  applying  information  security 
tools  across  multiple  boundaries  of  the  DoD  enterprise,  exploiting  a  vul¬ 
nerability  at  the  interior  of  the  network  is  increasingly  difficult. 

In  the  cyberspace  domain,  exploiting  a  system  can  be  categorized  as 
targeted  or  indiscriminate.  Indiscriminate  attacks  are  those  not  focused 
at  a  particular  entity;  rather,  they  seek  to  exploit  security  vulnerabilities 
across  many  systems.  These  attacks  are  often  thwarted  by  several  layers 
of  the  DoD  enterprise  network  security  as  the  level  of  system  fingerprint¬ 
ing  and  malware  complexity  is  limited  and  easily  recognized.  On  the 
other  hand,  a  targeted  attack  is  executed  by  a  highly  skilled  individual(s) 
who  seek  to  attack  a  specific  system.  Because  the  target  is  specific,  the 
attacker  will  become  an  expert  on  its  network  architecture,  hardware 
and  software  components,  and  intrusion  safeguards. 

As  layers  of  network  defense  increase,  attack  sophistication  grows 
as  well.  According  to  an  October  2011  report  released  by  the  U.S. 
Government  Accountability  Office  (GAO),  20  federal  agencies  reported 
an  increase  in  the  amount  of  targeted  and  indiscriminate  cyber  attacks 
against  critical  assets.  In  fact,  these  agencies  (one  of  which  was  DoD) 
reported  a  25  percent  increase  in  the  number  of  reported  intrusions  from 
2009  to  2010  (GAO,  2011).  Unlike  a  medieval  castle  where  an  enemy  can 
defeat  a  single  layer  of  defense  without  compromising  the  entire  castle, 
cyber  security  is  defeated  if  a  single  available  attack  vector  is  success¬ 
fully  identified  and  exploited. 
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In  November  of  2007,  the  DoD  established  the  DoD  Information 
Assurance  Certification  and  Accreditation  Process  (DIACAP)  policy, 
captured  in  Department  of  Defense  Instruction  (DoDI)  8510.01  (DoD, 
2007).  The  purpose  of  DIACAP  is  to  provide  a  risk  management  pro¬ 
cess  for  IA  and  detail  IS  certification  and  accreditation  requirements 
throughout  a  system’s  life  cycle.  It  provides  a  step-by-step  process  to 
assure  DoD  systems  are  protected  and  defended  “by  ensuring  their 
availability,  integrity,  authentication,  confidentiality,  and  nonrepudia¬ 
tion”  (DoD,  2002).  DIACAP  was  created  out  of  necessity  as  the  former 
policy,  DITSCAP  (DoD  Information  Technology  Security  Certification 
and  Accreditation  Process),  was  ill-equipped  to  handle  information 
systems  in  the  net-centric  environment.  Improving  upon  DITSCAP, 
DIACAP  established  standardized  IA  controls,  a  schedule  to  review  an 
individual  system’s  IA  status,  and  testable  metrics  to  measure  security 
effectiveness.  Although  this  is  seen  as  an  improvement  over  DITSCAP, 
DIACAP  has  flaws. 

DIACAP  measures  security  effectiveness  according  to  a  prescribed 
timeline  (every  1  to  2  years).  Should  a  new  vulnerability  be  discovered, 
verification  of  a  security  patch  installation  could  then  take  months  before 
the  next  IA  inspection.  Furthermore,  the  IA  controls  monitor  known 
system  vulnerabilities  and  do  not  take  into  account  threat  monitoring, 
incident  detection,  or  incident  response.  DIACAP  is  a  risk  mitigation 
process  that  is  more  reactive  than  proactive  when  it  comes  to  system 
vulnerabilities.  It  works  well  for  new  IS  acquisitions  as  they  are  tested 
against  the  latest  vulnerability  database  with  the  latest  tools.  As  systems 
mature,  DIACAP  becomes  less  effective  as  threat  monitoring  takes  a 
back  seat  to  operations.  Currently,  efforts  are  underway  to  revise  how 
the  DoD  handles  certification  and  accreditation  of  its  systems.  These 
efforts  are  resulting  in  a  revision  of  the  DoDI  8500.02  series,  which  will 
mandate  the  use  of  the  DoD  Information  Assurance  Risk  Management 
Framework  (DIARMF).  While  DIARMF  addresses  many  shortcomings, 
it  will  be  years  before  the  process  is  fully  implemented. 

Penetration  testing,  or  authorized  hacking,  is  designed  to  evalu¬ 
ate  the  vulnerability  of  a  system  to  indiscriminate  and  targeted  cyber 
attacks.  The  goal  of  penetration  testers  is  to  obtain  unauthorized  privi¬ 
leges  by  exploiting  flaws  in  system  design  or  implementation  (Chairman 
of  the  Joint  Chiefs  of  Staff  Instruction  [CJCSI]  6510.01, 2011).  Other  inci¬ 
dents  that  penetration  testing  detects  include  denial  of  service,  malware 
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infection,  and  malicious  code.  Unfortunately,  penetration  testing  can 
never  prove  a  system  is  void  of  vulnerabilities.  Penetration  testing  only 
identifies  the  presence  of  known  vulnerabilities. 

Following  the  fielding  decision  for  new  information  systems,  orga¬ 
nizations  schedule  periodic  red  and  blue  team  penetration  exercises  to 
test  system  security.  These  tests  prove  effective  across  the  entire  DoD 
network;  however,  team  manpower  makes  it  difficult  to  assess  the  major¬ 
ity  of  systems.  In  an  effort  to  offset  the  manpower  shortfall,  the  DoD  is 
embarking  on  the  development  of  several  “cyber  test  ranges”  to  simulate 
real-world  conditions  in  a  controlled  environment.  Two  such  environ¬ 
ments  in  development  are  known  as  the  DoD  Information  Assurance 
Range  and  the  National  Cyber  Range. 

The  assemblage  of  the  DoD  defense-in-depth  strategy— DIACAP 
framework,  penetration  test  tools,  and  cyber  test  ranges— represents 
the  government’s  dedication  to  identify  known  system  vulnerabilities. 
Even  with  these  monumental  fiscal  and  personnel  investments,  the  DoD 
remains  incapable  of  measuring  the  security  of  a  system  with  a  mean¬ 
ingful  metric. 


Vulnerability  Markets 

Prior  to  1997,  the  Federal  Acquisition  Regulation  (FAR)  prohibited 
use  of  auctions  to  establish  contracts  between  the  government  and  sup¬ 
plier.  Language  in  the  FAR  specifically  prohibited  auction  techniques 
that  indicate  to  an  offeror  a  cost  that  it  must  meet  to  obtain  further 
consideration;  advise  an  offeror  of  its  price  standing  relative  to  another 
offeror;  and  otherwise  furnish  information  about  other  offerors’  prices 
(General  Services  Administration  [GSA],  2005,  pt.  15.610[e][2]).  In  1997, 
the  FAR  was  rewritten,  and  the  Office  of  Management  and  Budget  (OMB) 
removed  the  ban  on  government  involvement  in  auctions.  Ever  since, 
DoD  has  taken  advantage  of  the  e-commerce  auction  marketplace  to 
procure  a  variety  of  supplies.  Some  examples  of  DoD  auction  procure¬ 
ments  include: 

•  Navy  procuring  aircraft  and  ship  parts; 

•  Army  purchasing  IBM  ThinkPads,  saving  40  percent  off 
the  GSA  price; 
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•  Army  purchasing  spare  parts  for  the  Patriot  Missile  system; 
and 

•  Air  Force  acquiring  computer  equipment,  saving  27  percent. 

Additionally,  the  OMB  reported  that  the  Environmental  Protection 
Agency  conducted  94  reverse  auctions  in  2007  and  saved  almost  14  per¬ 
cent  from  the  government  estimate  (OMB,  2008).  In  tight  fiscal  times, 
where  saving  money  is  the  lifeblood  of  any  program,  the  savings  achieved 
by  using  online  auctions  are  hard  to  ignore.  Although  these  auctions  have 
only  been  employed  for  the  procurement  of  physical  items,  the  model  is 
applicable  toward  purchasing  software  security  vulnerabilities  in  the 
cyber  domain. 

Vulnerability  Market  Examples 

The  VM  emerged  as  a  way  for  security  researchers  and  hackers  to 
disclose  vulnerabilities  for  financial  gain.  In  the  past  decade,  three  VM 
models  surfaced,  which  form  the  majority  of  vulnerability  events:  the 
bug  challenge,  the  bug  bounty,  and  the  bug  auction. 

Bug  Challenge 

In  a  bug  challenge,  the  simplest  of  the  VM  models,  a  vendor  offers 
a  reward  for  reporting  vulnerabilities  related  to  a  particular  product. 
Unlike  the  other  two  models  described  in  this  section,  the  bug  chal¬ 
lenge  is  administered  directly  by  the  vendor  and  has  no  intermediary 
acting  as  a  clearinghouse.  This  model  has  a  couple  of  major  flaws.  First 
of  all,  prizes  for  a  vulnerability  are  not  market-driven  and  may  not  accu¬ 
rately  reflect  its  actual  value  (Schwalb,  2007).  As  finding  vulnerabilities 
involves  a  significant  investment,  researchers  could  sell  their  finds  on  the 
black  market  for  a  much  higher  price.  Secondly,  bug  challenges  are  often 
by  invitation-only,  where  the  researchers  are  placed  on  contract  and 
required  to  sign  nondisclosure  agreements.  By  restricting  the  research¬ 
ers,  the  vendors  have  the  ability  to  keep  any  vulnerabilities  secret  and 
subsequently  refuse  to  patch  the  products. 

For  3  weeks  in  2000,  the  Secure  Digital  Music  Initiative  (SDMI) 
conducted  a  public  bug  challenge  aimed  at  breaking  SDMI  watermarking 
technologies.  The  challenge  was  invitation-only  and  offered  a  cash  prize 
for  any  team  that  could  win  any  of  the  six  challenges  posed.  The  ultimate 
goal  was  to  identify  an  authentic  copy  of  the  audio  file  to  combat  online 


471 


Defense ARJ,  January  2014,  Vol.  21  No.  1: 466-484 


Strengthening  DoD  Cyber  Security  with  the  Vulnerability  Market 


music  piracy.  This  event  was  sanctioned  by  the  music  recording  industry 
and  required  all  participants  to  sign  a  nondisclosure  agreement  prior  to 
accessing  SDMI  datafiles  (Craver,  2001). 

Bug  Bounty 

Differing  from  a  bug  challenge,  a  bug  bounty  is  conducted  by  a  vendor 
seeking  to  pay  researchers  to  identify  malicious  code  used  to  infiltrate 
their  systems.  The  goal  of  this  market  model  is  for  a  vendor  to  flush  out 
an  undetected  vulnerability  currently  being  exploited  by  hackers.  Placing 
a  bounty  on  vulnerabilities  is,  by  nature,  a  reactive  countermeasure  to 
unsecure  software.  Recognizing  the  benefit  of  this  model,  the  company 
that  developed  the  popular  Web  browser  Mozilla  instituted  the  Mozilla 
Security  Bug  Bounty.  Starting  in  2004,  the  Bug  Bounty  sought  to  reward 
individuals  who  reported  critical  security  bugs  (The  Mozilla  Foundation, 
n.d.).  Since  December  of  2010,  Mozilla  has  paid  out  a  total  of  $104,000 
for  64  qualifying  bugs. 

Bug  Auction 

A  bug  auction  utilizes  auction  theory  to  conduct  a  VM.  Conducted 
in  an  online  environment,  sellers  of  vulnerabilities  attempt  to  maximize 
profit  while  buyers  attempt  to  minimize  cost.  In  bug  auctions,  two  models 
are  commonly  used:  the  English  and  Dutch  auctions,  described  in  Table  1. 


TABLE  1.  DESCRIPTION  OF  COMMON  AUCTION  TYPES 


Auction 

Type 

Bidding/ 

Offer  Process 

Description 

English 

(Traditional) 

Bids  increase 

This  is  the  typical  auction  in  which  a 
single  seller  of  a  single  item  (or  lot  of 
items)  receives  increasing  bids  from 
prospective  buyers.  The  auction  ends  at  a 
predetermined  time,  and  the  item  goes  to 
the  highest  bidder  for  the  highest  bid  price. 

Dutch 

(Reverse) 

Offers 

decrease 

The  exact  opposite  of  the  English  auction. 

A  single  buyer  of  a  single  item  (or  lot  of 
items)  receives  decreasing  offers  from 
prospective  sellers.  The  auction  ends  at 
a  predetermined  time,  and  the  item  is 
purchased  from  lowest  offerer  for  the 
lowest  price. 

Note.  Adapted 
Evidence,”  by  B. 

from  “Auctions  in  Defense  Acquisition:  Theory  and  Experimental 
Linsterand  D.  Mullin,  Acquisition  Review  Quarterly,  Summer  2002,  p.214. 
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In  contrast  to  the  widely  used  English  auction,  Dutch  (Reverse) 
auctions  are  less  frequently  utilized.  Reverse  auctions,  consisting  of  one 
buyer  and  multiple  sellers,  are  occurring  more  frequently  in  government 
material  acquisitions.  While  not  yet  applied  to  information  security, 
several  federal  agencies  recognize  the  financial  benefit  of  market  com¬ 
petition  between  suppliers.  Several  cases  of  successful  reverse  auctions 
are  detailed  in  Table  2. 


TABLE  2.  HISTORIC  SAVINGS  FROM  COMMERCIAL  AND 
GOVERNMENT  REVERSE  AUCTIONS 


Procuring  Activity 

Item  Procured 

Cost  Savings 

%  Savings 

State  of  Pennsylvania 

Aluminum 

$170,000 

9% 

United  Technologies 

Circuit  Boards 

$32,000,000 

53% 

Owens  Corning 

Packing  Materials 

$7,000,000 

7% 

U.S.  Navy  (NAVCIP) 

Ejection  Seat 
Components 

$933,000 

28% 

U.S.  Air  Force 

Computers 

$88,000 

27% 

DESC 

Natural  Gas 

$972,000 

22% 

U.S.  Army  CECOM 

Transformers 

$195,000 

53% 

Note.  Adapted  from  CLC031:  Reverse  Auctioning  [Online  course  module],  published  by 
the  Defense  Acquisition  University,  2012.  NAVCIP  =  Naval  Inventory  Control  Point;  DESC 
=  Defense  Energy  Support  Center;  CECOM  =  Communications-Electronics  Command. 


Reverse  auctions  may  benefit  DoD  information  security  in  three 
ways.  First,  reverse  auctions  enhance  cyber  security  through  early  iden¬ 
tification  of  vulnerabilities.  Second,  the  auctions  leverage  the  skills  and 
knowledge  of  private  security  researchers  in  the  private  sector.  Third, 
when  compared  to  an  expected  loss,  executing  an  auction  costs  far  less 
than  remediating  an  attack. 

Based  on  these  advantages,  this  article  concentrates  on  developing  a 
reverse  auction  model  to  be  used  by  the  DoD  prior  to  full  system  deployment. 

Applying  Reverse  Auctions 

While  traditional  auctions  aim  to  increase  bids  on  an  item  for  sale, 
reverse  auctions  strive  for  the  opposite:  to  drive  prices  down.  In  reverse 
auctions,  buyers  initiate  the  auction  rather  than  the  seller.  The  buyers 
identify  a  product  or  service  they  want  to  buy  and  the  starting  price 
at  which  they  are  willing  to  compensate  the  sellers.  Once  the  auction 
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window  is  opened,  the  bidders  (e.g.,  the  sellers)  compete  to  offer  the  prod¬ 
ucts  or  services  at  the  lowest  cost  possible  while  still  retaining  a  profit. 
This  concept  takes  advantage  of  free  market  competition  to  lower  prices 
for  the  buyer  (Figure  1). 

FIGURE  1.  REVERSE  AUCTION-PRICE  DRIVEN  DOWN  OVER  TIME 


♦-  Bidder  #1 
■-  Bidder  #2 
Bidder#! 
•-  Bidder  #4 
Historic  Price 


The  purpose  of  using  a  reverse  auction  to  discover  vulnerabilities  is 
twofold.  The  first  objective  is  to  identify  possible  security  issues  associ¬ 
ated  with  a  software  product.  By  offering  cash  incentives,  vulnerability 
discovery  rates  increase  based  upon  the  number  of  researchers  attracted 
to  the  competition.  The  greater  the  number  of  researchers,  the  more 
likely  a  vulnerability  will  be  found.  The  second  objective  is  that  the  vul¬ 
nerability  auction  has  the  potential  to  provide  a  meaningful  metric  that 
would  describe  the  relative  security  of  a  product. 

Using  a  variant  of  the  reverse  auction  model  will  allow  the  govern¬ 
ment  to  use  auctions  for  the  procurement  of  software  vulnerabilities.  The 
government  (aka  the  buyer)  would  initiate  a  reverse  auction  within  an 
identified  pool  of  software  researchers  (aka  the  sellers).  The  government 
would  identify  and  provide  access  to  a  system  it  believes  to  be  secure. 
The  government’s  certainty  of  system  security  is  articulated  as  an  initial 
monetary  valuation,  expressed  as  the  variable  RQ.  The  objective  of  the 
researcher  participating  in  the  auction  is  to  disprove  the  government’s 
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assertion.  If  after  a  predetermined  amount  of  time  a  researcher  does  not 
report  a  vulnerability  to  the  government,  the  reward  value  increments 
from  RQ  to  Rr  In  the  Figure  2  example,  the  reward  first  increments  from 
R0  =  $10  to  R4  =  $15.  This  incremental  increase  repeats  until  a  vulner¬ 
ability  is  reported  or  until  the  prearranged  auction  window  closes.  Rn 
represents  the  amount  ($)  of  reward  at  increment  “n.”  If  a  researcher 
reports  a  software  vulnerability,  the  government  would  pay  the  current 
value  of  R  dollars.  The  Figure  2  example  shows  vulnerabilities  reported 
at  R2  and  R3  where  a  researcher  is  paid  $20  and  $25,  respectively.  At  the 
auction’s  conclusion,  the  last  value  of  the  reward  (R4),  equates  to  the 
security  of  the  system.  This  final  value,  or  the  Cost-To-Break  (CTB) 
metric,  is  the  amount  of  money  it  costs  an  individual  to  discover  and 
report  a  vulnerability. 


FIGURE  2.  REVERSE  AUCTION-REWARD  OVER  TIME,  UP  TO 
COST  TO  BREAK  (CTB) 


TIME 


Applying  VM  Concept  to  DoD  Information 
Systems  Acquisition 

For  the  DoD  VM  to  be  successful,  it  is  imperative  that  a  substantial 
set  of  qualified  software  researchers  participate.  As  arduous  as  it  is  to 
discover  software  vulnerabilities,  the  researchers  must  perceive  an 
adequate  level  of  compensation  for  their  efforts.  Compensation  to  incen- 
tivize  participation  can  take  many  forms  in  the  VM. 
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Financial  gain  is  the  most  common  type  of  incentive  offered  in 
commercial  VMs.  In  March  2012,  Forbes  published  a  price  list  that 
enumerates  the  financial  value  an  open  market  vulnerability  possesses 
(Table  3).  The  value  of  these  vulnerabilities  is  a  function  of  a  free-market 
economy  and  the  forces  of  supply  and  demand.  While  the  vulnerability 
may  not  be  worth  the  cost  to  the  vendor,  potential  consumers  of  vulner¬ 
abilities  may  perceive  the  cost  offsets  their  risk  and  any  potential  costs 
of  using  the  vulnerability. 


TABLE  3.  PRICE  LIST  FOR  SOFTWARE  VULNERABILITIES 


Application 

Vulnerability  Price  List 

Adobe  Reader 

$5,000  -  $30,000 

MAC  OSX 

$20,000  -  $50,000 

Android 

$30,000  -  $60,000 

Flash  or  Java  Browser  Plug-ins 

$40,000  -  $100,000 

Microsoft  Word 

$50,000  -  $100,000 

Windows 

$60,000  -  $120,000 

Firefox  or  Safari 

$60,000  -  $150,000 

Chrome  or  Internet  Explorer 

$80,000  -  $200,000 

iOS 

$100,000  -  $250,000 

Note.  Adapted  from  "Shopping  for  Zero-Days:  A  Price  List  For  Hackers’  Secret  Software 
Exploits,”  by  A.  Greenberg,  2012,  Forbes. 


To  establish  a  financial  reward,  the  DoD  must  provide  additional 
reassurances  in  the  form  of  nonattribution  and  anonymity  to  the 
researchers.  Nonattribution  and  anonymity  have  a  value  unto  them¬ 
selves.  By  offering  a  safe  and  nonattribution  environment,  security 
researchers  are  welcome  to  hack  a  government  system  without  threat 
of  being  prosecuted  under  state  and  federal  law.  These  reassurances, 
coupled  with  a  financial  reward,  must  counterbalance  the  price  of  a 
vulnerability  on  the  open  market. 

In  the  world  of  vulnerability  discovery,  a  major  motivation  amongst 
researchers  is  their  reputation.  In  the  hacker  community,  an  individual’s 
reputation  ranges  from  the  lowest  revered  status  of  “script  kiddie”  to 
the  highest  “elite”  status.  John  Arquilla,  a  professor  of  defense  analysis 
at  the  U.S.  Naval  Postgraduate  School  in  Monterey,  California,  recently 
estimated  that  only  around  100  “elite”  hackers  are  in  the  world  today 
(Carroll,  2012).  By  leveraging  reputational  exclusivity  and  the  egos  of 
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security  researchers,  the  DoD  could  incentivize  individuals  to  partici¬ 
pate.  A  researcher’s  reputation  may  be  elevated  based  upon  the  number 
of  vulnerabilities  or  new  attack  vectors  discovered.  A  heightened  reputa¬ 
tion  will  enhance  the  researcher’s  status  in  the  hacker  community  and 
could  also  result  in  job  and  consulting  offers  within  industry. 

Altruism,  in  the  cyber  security  environment,  is  also  a  powerful 
motivator.  It  is  so  powerful,  in  fact,  that  the  term  “white  hat”  hacker  was 
developed  specifically  for  the  altruistic  security  movement.  The  term 
white  hat  describes  a  hacker  ethically  opposed  to  the  abuse  of  IT  and  con¬ 
cerned  with  improving  overall  security  to  benefit  society.  Traditionally 
identified  as  specialists  in  penetration  testing  or  vulnerability  inves¬ 
tigation,  white  hats  use  their  expertise  to  protect  computer  health  and 
improve  system  security.  After  discovering  a  vulnerability,  white  hats 
will  either  contact  the  vendor  directly  to  force  a  patch  or  disclose  the 
vulnerability  to  a  third  party  like  the  United  States  Computer  Emergency 
Readiness  Team.  These  incentives,  with  cash  rewards  resulting  from  a 
DoD-sponsored  VM,  have  the  propensity  to  increase  software  vulner¬ 
ability  discovery  rates  and  software  security. 

Cost  to  Break 

Complete  product  security  is  almost  impossible  to  measure.  Metrics, 
such  as  SLOC,  can  describe  complexity  of  the  system,  but  fail  to  describe 
overall  security.  The  number  of  vulnerabilities  patched  over  a  given 
amount  of  time  is  also  a  useful  metric  that  is  quantifiable  and  easily 
understood.  Moreover,  a  company  can  advocate  the  amount  of  effort  (in 
dollars  and  time)  spent  securing  a  product.  The  failure  of  this  metric  is 
that  a  hacker  only  needs  a  single  undiscovered  vulnerability  to  exploit 
the  system.  To  provide  a  meaningful  way  of  measuring  the  security  of  a 
system,  the  DoD  requires  a  metric  that  is  quantifiable,  easily  understood, 
dynamic,  and  supports  IT  acquisition  milestones  for  decision  makers. 

The  traditional  definition  of  a  system’s  CTB  is  the  cost  that  an 
attacker  will  incur  in  compromising  the  system.  These  costs  may  include 
money,  research  time,  risk  of  being  caught,  etc.  Because  many  of  these 
costs  truly  vary  amongst  individuals,  calculating  this  view  of  the  CTB 
metric  is  unfeasible.  Rather  than  attempting  a  CTB  metric  focused  on  the 
individual,  this  article  proposes  using  the  VM  to  evaluate  the  security  of 
the  system  by  using  a  large  sample  population  of  security  researchers. 
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Using  a  VM  to  calculate  the  CTB  of  a  system  was  originally  pro¬ 
posed  by  Dr.  Stuart  Schechter  of  Harvard  University.  In  Dr.  Schechter’s 
model,  the  CTB  is  the  result  of  the  market  price  to  discover  system 
defects  governed  by  the  presence  of  competition  amongst  research¬ 
ers  (Schechter,  2002).  Otherwise  stated,  the  market-focused  CTB  is  a 
product  of  a  vulnerability  auction  where  an  IT  producer  offers  a  cash 
prize  to  free-market  researchers  to  break  their  system.  This  strategy 
of  paying  researchers  to  break  their  systems  is  used  frequently  today; 
however,  it  is  not  tracked  as  a  true  metric.  For  example,  since  2007  the 
CanSecWest  security  conference  has  hosted  the  annual  Pwn20wn  bug 
challenge,  which  rewards  researchers  for  hacking  into  some  of  the  most 
popular  computer  applications.  During  the  2013  Pwn20wn  challenge, 
researchers  were  awarded  $480,000  for  cracking  applications  developed 
by  Microsoft,  Google,  Adobe,  Mozilla,  and  Oracle.  Even  more  impres¬ 
sive,  Google  claimed  theirs  was  the  most  secure  operating  system  on  the 
market  by  offering  $110,000  for  a  browser  or  system-level  compromise 
delivered  via  a  Web  page.  At  the  end  of  the  conference,  the  entire  Google 
prize  pot  of  $3.14  million  remained  intact  (Thomson,  2013). 

The  inability  of  researchers  attending  the  conference  to  crack  the 
application  effectively  placed  the  CTB  metric  for  the  Google  Chrome  OS 
at  $110,000.  Accordingly,  this  metric  could  be  used  by  Google  to  compare 
its  security  to  other  operating  systems  (e.g.,  Windows,  Linux).  This  abil¬ 
ity  to  compare  applications  is  the  real  value  of  the  CTB  metric;  the  vendor 
is  now  able  to  highlight  the  security  of  its  product  relative  to  its  competi¬ 
tors.  For  a  discerning  consumer  concerned  with  product  security,  the 
CTB  may  influence  the  decision  to  purchase  one  product  versus  another. 
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The  CTB  metric  may  play  a  role  in  the  DoD  as  well.  Prior  to  awarding 
a  contract  to  a  specific  vendor,  the  DoD  establishes  a  source  selection 
strategy  or  acquisition  plan  that  outlines  all  evaluation  factors  affecting 
contract  award.  Should  software  security  be  an  evaluation  factor  in  the 
selection,  the  CTB  would  be  invaluable  in  the  comparison  of  multiple 
vendors.  The  hope  would  be  that  the  DoD  acquires  secure  software 
systems  prior  to  contract  award.  Additionally,  use  of  the  CTB  metric 
could  be  included  in  the  Joint  Capabilities  Integration  and  Development 
System  requirements  process.  By  requiring  that  an  IS  must  meet  speci¬ 
fied  thresholds,  the  contractor  and  government  ensure  the  IS  is  secure 
prior  to  deployment. 

Application  of  a  VM  leads  to  several  benefits.  First  of  all,  a  VM 
provides  an  additional  round  of  development  and  operational  testing. 
Second,  the  VM  increases  analysis  prior  to  fielding.  Increased  scrutiny 
and  additional  researchers  also  increase  the  vulnerability  disclosure 
rate  and  result  in  reducing  the  total  cost  of  ownership.  Third,  by  wide 
use  of  the  VM  to  enumerate  the  CTB  metric,  the  government  will  be  able 
to  compare  and  discern  multiple  systems. 

Conclusions  and  Recommendations 

Perfect  information  security  will  never  be  achieved.  Whether  vul¬ 
nerabilities  are  due  to  mistakes  by  the  software  developer,  a  vendor’s 
unwillingness  to  fix  flaws,  or  an  error  by  the  user,  the  outcome  is  the 
same— valuable  information  is  susceptible  to  attack.  In  the  informa¬ 
tion  age,  industry  understands  the  issues  of  software  vulnerability 
prevalence  as  much  as  the  DoD.  In  the  past  decade,  dozens  of  VMs  have 
sprung  into  existence  based  upon  the  perceived  need  to  enlist  nonorganic 
researchers  to  report  application  vulnerabilities.  The  responsibility  for 
securing  data  does  not  lie  solely  with  the  vendor  or  with  the  product 
consumer.  True  information  security  and  management  of  the  risk  of 
unauthorized  disclosure  is  the  responsibility  of  the  entire  community. 

Because  a  government  online  reverse  auction  market  for  the  purpose 
of  identifying  software  vulnerabilities  has  never  been  applied  to  a  DoD  IS 
acquisition,  concerns  arise  that  this  concept  is  legally  and  economically 
unfeasible.  Legally,  federal  statute  permits  and  encourages  the  use  of 
online  marketplaces  (GSA,  2005,  pts.  1.102, 4.5)  for  systems  acquisition. 
Furthermore,  precedent  in  the  commercial  and  government  sectors  is 
established.  As  reported  by  the  Washington  Post,  the  National  Security 
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Agency  (NSA)  allegedly  spent  more  than  $25  million  in  2012  to  pro¬ 
cure  vulnerabilities  (Fung,  2013).  With  respect  to  security  concerns,  the 
National  Institute  of  Standards  and  Technology  encourages  acquiring 
systems  that  are  “secure  by  design”  rather  than  those  that  are  “secure  by 
obscurity.”  While  obscurity  and  controlling  open  visibility  into  systems 
design  might  delay  potential  adversaries,  hidden  vulnerabilities  may 
ultimately  be  exploited  to  their  advantage.  Security  by  design  does  not 
rely  on  hiding  vulnerabilities.  Instead,  vulnerabilities  are  eliminated  by 
secure  software  design  principles.  In  cases  where  a  critical  system  must 
be  controlled  and  disseminated  to  trusted  individuals,  entry  into  the  VM  is 
governed  through  the  enforcement  of  appropriate  clearance  requirements. 

Economically,  each  IS  vulnerability  has  the  probabilistic  potential  to 
cost  the  DoD  immense  resources.  Although  calculating  the  consequences 
of  using  a  system  with  unknown  vulnerabilities  is  difficult  to  quantify, 
discovery  of  a  vulnerability  prior  to  use  in  an  operational  environment  is 
more  cost-effective  than  remediating  it  postdeployment.  Decreasing  the 
probability  and  increasing  the  discovery  rate  of  system  vulnerabilities 
are  the  primary  goals  of  the  proposed  VM  model  for  DoD-acquired  sys¬ 
tems.  Not  only  will  the  discovery  of  an  unknown  vulnerability  effectively 
reduce  the  probability  of  a  successful  attack,  life-cycle  operations  and 
maintenance  costs  are  also  reduced.  Addition  of  a  VM  to  the  develop¬ 
ment  phases  within  DoD  acquisition  results  in  a  proactive  approach  to 
information  security  and  mission  assurance. 

Use  of  this  auction  model  will  create  a  meaningful  and  easily  under¬ 
standable  metric  to  ensure  the  DoD  acquires  systems  with  built-in 
security.  This  CTB  metric  has  the  propensity  to  reform  the  defense 
industrial  base  as  well  as  conform  to  information  security  requirements 
as  dictated  by  the  warfighter.  Through  the  mutual  cooperation  between 
industry  and  the  military  in  securing  information,  the  DoD  will  optimize 
security  investments,  secure  critical  information,  and  provide  an  effec¬ 
tive  and  resilient  warfighting  capability. 
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Conceptual  models  based  on  economic  and  operations 
research  principles  can  yield  valuable  insight  into  defense 
acquisition  decisions.  This  article  focuses  on  models 
that  place  varying  degrees  of  emphasis  on  each  objec¬ 
tive  of  the  defense  acquisition  system:  cost  (low  cost), 
schedule  (short  cycle  times),  and  performance  (high 
system  performance).  The  most  appealing  conceptual 
model  is  chosen,  which  the  authors  posit  that,  if  adopted, 
would  lead  to  shifts  in  priorities  that  could  facilitate 
better  outcomes,  as  empirical  results  suggest.  Finally, 
several  policy  prescriptions  implied  by  the  model  are 
briefly  explored. 


«  Image  designed  by  Diane  Fleischer 


Defense  ARJ,  January  2014,  Vol.  21  No.  1 : 486-504 


486 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Principles  from  microeconomic  theory  and  operations  research  can 
provide  insight  into  acquisition  decisions  to  produce  military  capabili¬ 
ties  in  an  environment  of  scarce  resources.  To  begin  a  discussion  of  the 
analytic  models  involved,  it  helps  to  identify  desired  outcomes.  In  the 
acquisition  literature,  cost  (low  cost),  schedule  (short  cycle  times),  and 
performance  (high  system  performance)  are  generally  regarded  as  primary 
objectives  in  fielding  new  systems  (Department  of  Defense  [DoD],  2006). 
This  article  focuses  on  conceptual  models  involving  these  three  goals  that 
place  varying  degrees  of  emphasis  on  each  objective.  The  most  appealing 
conceptual  model  is  chosen,  which  we  posit  that,  if  adopted,  could  clearly 
lead  to  badly  needed  shifts  in  priorities.  Empirical  results  bolster  the 
proposition  that  changing  priorities  could  lead  to  better  outcomes.  Finally, 
we  discuss  several  policy  changes  implied  by  the  chosen  model. 

In  pursuing  a  defense  acquisition,  the  immediate  question  becomes 
what  combination  of  cost,  schedule,  and  performance  can  or  should  be 
considered  optimal?  If  the  only  guidance  provided  to  analysts  is  to  do 
their  best  to  minimize  cost  and  time  to  field  while  maximizing  perfor¬ 
mance,  then  making  tradeoffs  will  rely  on  professional  military  judgment 
at  best  or  become  arbitrary  at  worst.  The  basis  for  decision  making  can 
be  unclear  and  result  in  disputes,  and  the  acquisition  professional  may 
lose  sight  of  the  overall  objective.  Therefore,  the  goal  of  this  article  is  to 
develop  a  new  way  of  characterizing  the  acquisition  problem  that  will 
help  decision  makers  make  more  informed  tradeoffs. 

The  Efficient  Frontier  of  Defense  Acquisition 

Each  defense  acquisition  program  can  be  judged  by  how  much  input 
(time  and  money)  is  consumed  to  produce  the  desired  military  out¬ 
put  (performance  or  capability).  This  uncontroversial  statement  of 
defense  acquisition  system  objectives  reveals  how  this  incredibly  com¬ 
plex  system  reduces  to  a  relatively  simple  problem  involving  production 
economics.  To  put  it  in  mathematical  terms,  a  system’s  performance  can 
be  expressed  hypothetically  as  a  function  of  the  independent  variables 
time  and  money,  as  described  in  the  following  associated  interactions. 

Performance  =/ (time,  money) 
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FIGURE  1.  NOTIONAL  PERFORMANCE  ISOQUANTS 


FIGURE  2.  THE  DEFENSE  ACQUISITION  EFFICIENT  FRONTIER 


Using  this  function,  performance  can  be  plotted  as  a  function  of 
either  time  or  money  in  a  standard  Cartesian  graph.  Another  way  to  visu¬ 
alize  this  function  is  to  hold  performance  constant  while  varying  time 
and  money.  All  of  the  combinations  of  time  and  money  result  in  the  same 
performance  makeup  of  what  is  called  a  performance  isoquant.  Figure  1 
illustrates  this  concept  with  two  notional  performance  isoquants.  If  cost 
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and  time  are  then  divided  by  performance  and  graphed  in  the  same  man¬ 
ner,  it  becomes  clear  that  increases  in  either  cost/performance  or  time/ 
performance  are  undesirable.  The  efficient  frontier  of  cost,  schedule,  and 
performance  represents  optimal  outcomes  under  the  current  system. 
The  curve  in  Figure  2  illustrates  the  efficient  frontier,  and  points  above 
the  line  are  considered  inefficient. 


The  goal  of  defense  acquisition  system  managers  and  practitioners 
would  be  to  choose  to  develop  and  produce  systems  that  occupy  the  effi¬ 
cient  frontier.  Although  reaching  the  efficient  frontier  may,  in  practice, 
be  extremely  challenging,  improvements  toward  that  end  are  constantly 
sought.  As  such,  a  commonly  occurring  issue  arises  when  one  considers 
how  a  hypothetically  inefficient  program  should  be  improved  to  reach 
efficiency— should  costs  be  cut,  time-to-launch  condensed,  and/or  per¬ 
formance  enhanced?  In  other  words,  how  should  marginal  resources 
be  allocated? 

If  the  system  is  allowed  to  function  with  limited  oversight,  how 
might  the  actors  behave  and  what  are  the  implications  of  their  decisions? 
What  might  be  a  better  way  to  state  the  acquisition  problem  to  develop 
a  commonsense  approach  to  the  question  of  how  to  make  tradeoffs 
between  cost,  schedule,  and  performance?  Finally,  do  empirical  results 
tend  to  support  or  refute  the  recommended  model,  and  what  sort  of 
actions  can  be  taken  to  make  the  current  defense  acquisition  system 
more  consistent  with  the  chosen  conceptual  model? 
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Economic  Incentives  of  Defense  Contractors 
and  the  Defense  Department 

Edmund  Conrow  (1995)  developed  an  excellent  microeconomic 
framework  to  investigate  the  incentives  of  buyers  and  sellers  in  the 
defense  acquisition  system.  His  most  important  insight  is  that  while  gov¬ 
ernment  and  contractors  have  objectives  that  can  and  often  do  conflict, 
negative  outcomes  associated  with  aligned  government  and  contractor 
incentives  are  quite  likely  to  occur— and  have  occurred  in  the  past.  To 
avoid  similar  outcomes  in  the  future,  mitigating  incentives  or  controls 
must  be  put  in  place. 

The  government  and  contractors’  preferences  can  be  discussed  in 
terms  of  cost,  schedule,  and  performance.  Clearly,  the  government  pre¬ 
fers  low  cost,  shorter  cycle  times,  and  high  performance.  Meanwhile, 
contractors’  preferences  are  more  nuanced.  In  the  absence  of  a  fixed- 
price  contract,  contractors  generally  prefer  rising  system  costs  to 
strengthen  their  own  bottom  lines.  However,  this  preference  is  not 
absolute  because  spiraling  costs  can  result  in  negative  reviews  on  the 
Contractor  Performance  Assessment  Reporting  system.  Developing 
higher  performing  systems  improves  a  contractor’s  future  market  posi¬ 
tion  by  keeping  it  on  the  cutting  edge  of  technology.  Finally,  extended 
schedules  imply  a  longer  relationship  with  a  government  client  that 
may  offer  future  system  development  opportunities,  though  this  incli¬ 
nation  can  backfire  when  taken  too  far,  as  with  preferring  high  system 
costs.  As  depicted  in  Conrow  (1995),  this  can  be  illustrated  graphically 
using  government  and  contractor  indifference  curves  sketched  in  cost- 
performance  space. 

Figure  3  provides  a  visual  representation  of  the  indifference  curves 
inspired  by  Conrow  (1995).  The  government’s  utility  curves  are  labeled 
UGi  where  increasing  i  indicates  increasing  utility.  For  the  government, 
utility  clearly  increases  from  right  to  left  and  bottom  to  top.  Meanwhile, 
contractor  utility  indifference  curves  are  labeled  analogously  by  UCi-  The 
technical  possibilities  curve  represents  what  performance  levels  are 
possible  at  each  cost  and  intersects  both  sets  of  indifference  curves  at 
two  places,  A  and  B.  These  are  feasible  and  efficient  choices  for  cost  and 
performance.  Meanwhile,  point  C  is  infeasible  due  to  lack  of  adequate 
technology,  while  point  D  is  inefficient  and,  thus,  inferior  to  points  A  and 
B.  Of  the  two  feasible  and  efficient  points,  point  B  dominates  because  it 
offers  both  government  and  contractor  a  higher  level  of  utility. 


Defense ARJ,  January  2014,  Vol.  21  No.  1: 486-504 


490 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


FIGURE  3.  GOVERNMENT  AND  CONTRACTOR  COST  AND 
PERFORMANCE  ISO-SCHEDULE  CURVES  WITH  TECHNICAL 
POSSIBILITY  CURVE 


COST 

This  economic  analysis  reveals  that,  in  the  absence  of  strong  cost 
control,  the  program  office  and  contractor  will  tacitly  conspire  to 
increase  performance  at  the  expense  of  cost.  That  is,  the  program  office 
and  the  supplier  can  agree  on  performance,  but  they  cannot  agree  on 
either  cost  or  schedule.  Therefore,  the  natural  inclination  is  to  maximize 
performance  and  see  how  much  cost  and  schedule  can  be  massaged. 
This  insight  from  economic  theory  reinforces  the  beliefs  of  defense 
experts.  Indeed,  since  program  managers  and  executive  officers  clearly 
benefit  from  managing  larger  and  more  complex  programs,  some  claim 
that  the  budget  and  schedule  objectives  are  not  as  important  in  the  eyes 
of  important  decision  makers.  Laws  like  Nunn-McCurdy  are  intended 
to  mitigate  this  tendency  to  let  costs  and  schedule  spiral  out  of  control. 
However,  some  argue  that  Nunn-McCurdy  has  failed  in  its  stated  goal 
because  it  only  requires  congressional  notification  of  a  breach  (Ewing, 
2011).  Often,  Congress  is  already  aware  of  the  program’s  struggles  and 
does  not  react  to  the  breach.  Further  steps  need  to  be  taken  to  effectively 
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discipline  the  system.  A  new  way  to  look  at  the  objectives  of  developing 
and  fielding  a  system  might  help  clarify  what  needs  to  be  done,  and  is  the 
focus  of  this  article’s  final  section. 

A  Constrained  Optimization  Approach 

Reconciling  the  competing  objectives  of  the  defense  acquisition 
system  has  proven  to  be  quite  challenging.  While  optimization  and 
tradespace  tools  are  useful,  senior  leaders  bear  primary  responsibility 
for  communicating  DoD  acquisition  strategy  through  their  statements 
and  actions.  A  notional  optimization  problem  specification  can  be  help¬ 
ful  in  framing  a  strategy  in  a  manner  that  is  internally  consistent,  and 
such  that  the  goals  (referred  to  as  the  objective  function  in  operations 
research)  and  constraints  reinforce  one  another. 

In  the  language  of  operations  research,  any  proposed  system  must 
not  exceed  a  maximum  budgeted  cost;  take  too  long  to  field  and,  thus, 
sacrifice  military  utility;  or  fall  below  minimum  performance  standards. 
In  mathematical  terms: 


Cost  <  C 
Time  <  t 
Performance  >  P 

As  in  the  previous  section,  these  constraints,  taken  in  isolation,  do 
not  offer  any  rationale  for  making  tradeoffs.  For  instance,  what  should 
be  done  if  the  system’s  development  schedule  slides  to  the  right— should 
costs  be  increased  by  allowing  more  overtime  or  hiring  more  staff,  or 
should  performance  goals  be  sacrificed  to  tighten  the  schedule?  Adoption 
of  a  decision  rule  can  help  facilitate  such  decisions.  Even  the  adoption 
of  a  heuristic  decision  rule  can  enlighten  analysts  and  prove  useful  in 
promoting  the  goals  of  the  acquisition  system.  That  is,  even  though  this 
conceptual  model  is  admittedly  a  greatly  simplified  representation  of  an 
extraordinarily  complex  system,  it  still  yields  valuable  insights  to  those 
in  defense  acquisition  leadership  positions.  Indeed,  the  models  discussed 
throughout  this  article  assume  decision  makers  possess  more  informa¬ 
tion  than  is  ever  available  in  practice,  so  much  is  left  to  professional 
judgment,  particularly  regarding  unforeseen  cost  and  schedule  risks. 
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This  set  of  constraints  suggests  a  constrained  optimization  prob¬ 
lem  specification  could  be  useful  in  framing  the  discussion.  Standard 
operations  research  principles  dictate  that  three  different  system  speci¬ 
fications  are  possible.  The  potential  explanatory  power  of  each  will  be 
judged  by  the  implications  arising  when  the  specification  is  translated 
into  English. 


Budget  Specification  Model 

Minimize  Cost 
subject  to  Performance  >P 
Time  <  t 

The  Budget  Specification  Model  can  be  considered  as  an  attempt 
at  obtaining  defense  “on  the  cheap.”  Its  appeal  is  intuitive,  particularly 
in  today’s  budgetary  environment  of  increasing  fiscal  constraints,  but 
this  specification  can  be  removed  from  consideration  because  the  cost 
uncertainty  associated  with  it  would  severely  disrupt  the  planning,  pro¬ 
gramming,  budgeting,  and  execution  process.  Budgets  must  be  specified 
precisely,  and  including  the  program  budget  as  a  constraint  is  the  only 
feasible  way  to  proceed. 

Capabilities-Based  Specification  Model 

Maximize  Performance 
subject  to  Cost  <  C 
Time  <t 

The  Capabilities-Based  Specification  Model  formalizes  the  undisci¬ 
plined  incentive  structure  outlined  in  the  previous  section.  Of  the  three 
approaches,  this  model  is  the  most  vulnerable  to  requirements  creep. 
While  this  specification  will  almost  certainly  lead  to  the  highest  per¬ 
forming  systems,  it  will  leave  the  DoD  vulnerable  to  former  Secretary  of 
Defense  Robert  Gates’  critique  of  “running  up  the  score”  (Thompson  & 
Karon,  2009).  Furthermore,  the  two  constraints  often  seem  to  be  some¬ 
what  negotiable.  Timelines  are  lengthened  to  lower  short-term  costs  and 
expenditure  becomes  important,  seemingly,  only  when  Nunn-McCurdy 
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limits  are  in  danger  of  being  breached.  As  previously  mentioned,  even  a 
Nunn-McCurdy  breach  often  fails  to  motivate  change.  In  other  words, 
the  constraints  are  not  binding,  leading  to  an  undisciplined  system.  If 
this  specification  is  chosen,  the  focus  must  be  on  disciplining  the  system 
through  enforcement  of  the  constraints  and  ending  requirements  creep, 
for  example.  With  primary  emphasis  on  maximizing  capability,  there 
will  always  be  a  strong  tendency  to  extend  timelines  to  get  that  last  bit 
of  performance. 


Threat  Specification  Model 

Minimize  Time 
subject  to  Performance  >  P 
Cost  <  C 

The  Threat  Specification  Model  best  reflects  the  thinking  behind 
amendments  to  the  Federal  Acquisition  Regulation  for  needs  of  “unusual 
and  compelling  urgency”  (General  Services  Administration,  2005).  The 
naming  of  the  threat  specification  arises  from  the  idealized  case:  when 
a  threat  is  imminent  and  a  near-term  response  is  required.  This  sort  of 
rapid  acquisition  authority  is  designed  to  meet  immediate  warfighter 
needs.  Some  experts  have  advocated  a  similar  system  be  adopted  for  all 
acquisition.  In  the  words  of  retired  Air  Force  Lieutenant  General  David 
Deptula,  “...we  need  to  be  able  to  operate  much  quicker  and  inside  our 
adversary’s  decision  rate”  (Hoffman,  2010). 

This  problem  specification  has  intuitively  appealing  implications- 
and  it  is  not  susceptible  to  Secretary  Gates’  “runnin’-up-the-score” 
criticism.  Using  time  as  the  objective  function  will,  in  most  cases,  rein¬ 
force  the  cost  constraint  somewhat  because  simplifying  the  acquisition 
process  will  undoubtedly  reduce  costs.  In  any  case,  in  nonemergency 
acquisition,  this  objective  function  and  its  associated  constraints  are 
in  harmony  to  a  greater  extent  than  the  capabilities-based  specifica¬ 
tion.  The  primacy  of  performance  in  the  capabilities-based  specification 
naturally  leads  to  higher  costs  and  longer  cycle  times,  while  enhanced 
focus  on  reducing  cycle  times  should  not  cause  overall  costs  to  rise  to 
the  same  degree  as  long  as  the  concept  is  not  taken  too  far.  The  DoD’s 
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Better  Buying  Power  2.0  initiative  (Kendall,  2012)  makes  the  mutually 
reinforcing  nature  of  reducing  both  cycle  times  and  cost  explicit  during 
the  development  phase: 

This  initiative  will  assess  the  root  causes  of  long  product  cycle 
time,  particularly  long  development  cycles,  with  the  goal  of  sig¬ 
nificantly  reducing  the  amount  of  time,  and  therefore  cost,  it 
takes  to  bring  a  product  from  concept  to  fielding,  (pp.  5-6) 

Furthermore,  the  capability  constraint  is  unlikely  to  be  violated,  or 
the  system  is  likely  to  be  cancelled  due  to  inadequate  military  utility. 
Because  of  the  objective  function’s  primacy,  requirements  creep  should 
not  be  as  ubiquitous.  Therefore,  this  approach  should  have  the  added 
benefit  of  being  easiest  to  discipline. 

An  immediate  critique  of  the  Threat  Specification  Model  is  the 
possibility  of  being  stuck  with  less  capable  systems  for  many  years.  If 
the  acquisition  cycle  is  shortened,  this  criticism  is  blunted  somewhat 
because  our  armed  forces  do  not  have  to  be  impaired  as  long  with  less 
capability,  but  it  still  remains  troubling.  The  most  plausible  solution 
is  a  hybrid  framework  that  follows  this  decision  rule,  but  retains  some 
of  the  virtues  of  the  existing  system.  While  the  chosen  model  does  not 
advocate  for  a  single-minded  focus  on  the  schedule,  program  managers 
should  avoid  roadblocks  that  might  stymie  a  program  in  the  absence  of 
overwhelming  cost  or  performance  benefits. 

Empirical  Results 

While  the  recommended  theoretical  framework  may  be  intui¬ 
tively  appealing,  the  more  important  question  is  whether  any  empirical 
evidence  exists  to  support  the  hypothesis  that  an  enhanced  focus  on 
schedule  will  improve  acquisition  outcomes  more  generally.  Since  the 
available  historical  data  are  merely  observational,  obviously  there  will 
be  no  perfect  test.  However,  the  results  of  a  few  statistical  tests  offer 
some  hope  of  improving  acquisition  outcomes  through  adopting  a  dif¬ 
ferent  mindset. 

A  potentially  contentious  argument  made  in  advancing  the  merits 
of  the  Threat  Specification  Model  described  in  the  previous  section,  was 
that  an  enhanced  focus  on  the  schedule  would  also  pay  dividends  by  help¬ 
ing  to  avoid  unpleasant  cost  outcomes.  This  proposition  can  be  examined 
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by  determining  whether  schedule  problems  precede  or  occur  simultane¬ 
ously  with  cost  overruns.  If  schedule  slippages  occur  first,  this  suggests 
that,  in  some  cases,  cost  overruns  could  be  prevented  by  first  ensuring 
a  program  can  meet  its  schedule.  Of  the  54  programs  listed  as  incurring 
Nunn-McCurdy  cost  breaches  on  the  Defense  Acquisition  Management 
Information  Retrieval  system  from  December  1997  to  June  2012,  38 
(or  70.4  percent)  experienced  Acquisition  Program  Baseline  schedule 
breaches  in  a  previous  Selected  Acquisition  Report.  If  the  timing  of  the 
breaches  is  determined  by  chance,  schedule  breaches  should  take  place 
before  cost  breaches  about  50  percent  of  the  time.  If  we  consider  these 
programs  to  be  representative  of  all  programs,  this  proposition  can  easily 
be  tested  statistically  using  a  simple  exact  binomial  test.  Carrying  out 
such  a  test  reveals  that  the  observed  results  would  be  very  unlikely  to 
occur  due  to  chance  (p-value  <  0.001),  so  the  unavoidable  conclusion  that 
schedule  slippages  more  often  precede  cost  breaches  than  the  converse 
is  true.  If  the  same  proposition  is  tested  while  allowing  for  simultane¬ 
ous  breaches,  the  results  are  even  more  conclusive— 43  of  54  schedule 
breaches  occurred  before  or  simultaneously  with  cost  breaches  (p-value 
<  l.OxlO'5).  These  test  results  are  presented  in  Table  1.  Such  outcomes 
strongly  suggest  that  schedule  problems  precede  cost  breaches.  Therefore, 
preventing  the  schedule  breach  could  eliminate  the  cost  breach  too. 


TABLE  1.  TIMING  OF  ACQUISITION  PROGRAM  BASELINE 
SCHEDULE  BREACHES  VS.  NUNN-MCCURDY  COST  BREACHES 


Criteria 

Frequency 

Schedule  Breach  First 

70.4%* 

Schedule  Breach  First/Same  Time 

79.6%* 

*  -  p-value  <  O.Ol,  n  =  54 


The  next  question  that  arises  is  whether  focusing  on  schedule  can 
also  benefit  performance  outcomes.  This  is  much  more  difficult  to  test 
since  performance  goals  must  clearly  be  met  within  a  certain  time- 
frame— making  the  two  goals  virtually  indistinguishable.  The  key  here 
is  to  reverse  the  direction  of  causation  and  set  performance  goals  that 
are  realistically  within  reach  with  existing  technology  or  imminent 
scientific  advancement.  That  is,  it  is  important  to  avoid  reaching  for 
unrealistic  technological  advances  to  achieve  higher  performance 
goals.  This  proposition  has  been  supported  empirically  through  analy¬ 
sis  of  Selected  Acquisition  Reports  and  Defense  Acquisition  Executive 
Summaries  (Dacus,  2012).  Before  DoD  acquisition  Milestone  B,  each 
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critical  technology  element  of  a  program  is  assessed  to  determine  how 
far  along  the  discovery  process  each  element  has  progressed.  Low  tech¬ 
nology  readiness  levels,  which  describe  the  maturity  of  these  critical 
technology  elements,  have  been  linked  to  larger  cost  overruns  and  more 
pronounced  schedule  slippage.  If  two  simple  decision  rules  that  require 
minimum  individual  and  system-wide  technology  readiness  levels 
before  a  development  effort  is  allowed  to  become  a  program  of  record  are 
enforced,  cost  and  schedule  outcomes  are  improved  dramatically,  with 
mean  differences  in  outcomes  that  are  statistically  significant  (Table  2). 
These  results  provide  strong  evidence  that  setting  more  realistic  perfor¬ 
mance  goals  can  lead  to  substantial  improvements  in  cost  and  schedule 
performance.  Therefore,  performance  goals  should  be  set  with  the  schedule 
very  much  in  mind. 

TABLE  2.  APPLICATION  OF  THE  MINIMUM  TECHNOLOGY 


READINESS  LEVELS  DECISION  RULE 

Quantity  of  Interest 

No  Violation 

Violation 

Mean  Months  Behind  Schedule** 

7.7  mos. 

31.2  mos. 

Mean  Percentage  Cost  Overrun* 

3.2% 

35.5% 

*  p-value  <  0.05,  **  p-value  <  0.01,  n  =  50. 

Note.  Adapted  from  "Improving  Acquisition 

Outcomes  Through 

Simple  System 

Technology  Readiness  Metrics,"  by  C.  Dacus,  2012,  Defense  Acquisition  Research 
Journal,  79(4),  p.  453. 


Some  Policy  Recommendations 

The  Threat  Conceptual  Model  (also  referred  to  as  the  Threat 
Specification  Model)  lends  itself  to  interpretation  through  various  policy 
recommendations.  Some  of  these  recommendations  and  a  discussion 
of  their  implications  follow.  These  examples  capture  the  spirit  of  the 
Threat  Conceptual  Model,  but  do  not  represent  an  attempt  to  catalog 
all  of  its  implications.  Creative  implementation  of  the  values  inherent 
in  the  model  may  lead  to  unanticipated  benefits.  With  that  caveat  in 
mind,  schedules  could  be  shortened  by  focusing  on  stability,  by  avoiding 
technology  overreach,  by  keeping  systems  as  simple  as  possible,  and  by 
initiating  fewer  programs. 
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Insist  on  Stability 

Continual  change,  whether  in  the  requirements  themselves  or  in 
available  program  funding,  leads  to  extended  schedules  and  should  be 
minimized  to  emphasize  meeting  the  program  schedule.  Requirements 
instability  can  stem  from  a  desire  to  improve  performance  above  what 
was  originally  envisioned,  but  consistently  engaging  in  requirements 
creep  conflicts  with  the  proposed  mindset.  According  to  those  surveyed 
for  the  2006  Defense  Acquisition  Performance  Assessment  Report, 
requirements  instability  was  the  most  commonly  identified  reason  for 
cost  growth  and  schedule  extensions  (Kadish,  2006).  According  to  a 
2008  Government  Accountability  Office  report,  Defense  Acquisitions: 
Assessments  of  Selected  Weapon  Programs,  cost  increases  for  programs 
with  no  requirements  changes  after  system  development  were  signifi¬ 
cantly  lower  (72  percent  vice  11  percent).  Further,  according  to  a  2011 
GAO  study,  requirements  changes  added  5  months  to  delays  already 
being  experienced  (GAO,  2011).  Intuitively,  this  makes  sense;  changing 
direction  after  the  program  is  underway  is  likely  to  increase  the  time  to 
field  and  raise  costs. 


According  to  those  surveyed  for  the  2006 Defense 
Acquisition  Performance  Assessment  Report, 
requirements  instability  was  the  most  commonly 
identified  reason  for  cost  growth  and  schedule 
extensions  (Kadish,  2006). 


Just  as  adding  requirements  tends  to  cause  a  later  delivery  of  prom¬ 
ised  capability,  so  does  a  slip  in  a  program’s  budget.  This  often  stems  from 
a  desire  to  reduce  costs  in  the  short  run  or  from  shifting  priorities.  Of 
course,  some  budget  changes  may  be  unavoidable  due  to  congressional 
action,  but  the  Services  must  make  a  commitment  to  the  schedule  and 
exercise  discipline  when  possible  to  avoid  disrupting  the  program.  If 
requirements  are  to  be  achievable,  there  must  be  the  financial  resources 
to  pay  for  them.  As  it  stands,  every  year  programs  are  forced  to  rejustify 
themselves  to  virtually  every  funding  authority  within  the  program’s 
funding  hierarchy.  During  that  process,  Congress  may  decide  to  adjust 
the  procurement  quantity,  as  it  did  with  the  C-17  when  additional 
numbers  were  added  to  the  planned  buy,  creating  some  second-  and 
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third-order  effects  to  other  programs.  More  often.  Congress  decides 
to  cut  the  number  of  program  units,  as  was  the  case  with  the  F-22. 
Unfortunately,  reduced  quantity  increases  the  per-unit  cost  because  the 
sunk  development  costs  are  allocated  across  fewer  units  (Deloitte,  2009). 

Shifting  requirements  and  budget  instability  often  work  together  to 
lengthen  programs.  Deloitte  (2009)  noted  that: 

...most  programs  are  funded  and  launched  while  there  is  still 
significant  uncertainty  about  everything  from  systems  and 
technologies  to  integration,  interoperability,  and  supply  chain 
requirements.  This  lack  of  certainty  and  knowledge  makes  it  dif¬ 
ficult  or  impossible  to  make  informed  funding  decisions,  which 
often  leads  to  cost  overruns  and  schedule  delays,  (p.  12) 

The  DoD  begins  programs  with  too  many  unknowns  leading  to  longer 
cycles,  greater  costs,  and  more  oversight. 

Avoid  Technology  Overreach 

Study  after  study  notes  that  the  DoD  reaches  for  capabilities  that 
are  too  far  off— pushing  for  “exquisite”  solutions  as  former  Secretary 
of  Defense  Gates  called  them.  These  capabilities  are  often  technol¬ 
ogy-driven  and  far  beyond  current  reach.  The  GAO  reports  too  many 
programs  enter  into  one  phase  or  another  without  mature  technologies 
(GAO,  2011).  As  mentioned  earlier  in  the  Empirical  Results  section, 
immature  technology  at  both  the  subsystem  and  system  levels  leads  to 
delayed  delivery  of  a  promised  capability  to  the  warfighter,  and  to  budget- 
busting  cost  overruns.  The  longer  the  program  runs,  the  greater  the 
temptation  to  add  to  it  the  latest  technology  or  some  other  requirement, 
placing  the  program  in  what  Spinney  called  a  “death  spiral  development” 
(Fallow,  2002).  A  more  disciplined  approach  to  selecting  the  right  tech¬ 
nologies  for  a  proposed  system  will  enable  program  managers  to  have 
more  effective  control  over  schedules  and,  therefore,  cost  (Ward,  2010). 

Keep  It  Simple 

Complexity  is  also  a  factor  that  drives  long  acquisition  cycles  to 
push  for  increased  performance  at  the  expense  of  the  schedule.  It  might 
be  prudent  to  look  at  less  complex  systems  that  are  not  “silver  bullet 
systems”  capable  of  being  all  things  to  all  users,  but  instead  incorporate 
“ready-for-prime-time”  technology.  For  example,  the  F-22  was  jokingly 
referred  to  as  the  E/F/A/B/C/K-22  to  indicate  the  DoD  was  counting  on 
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one  aircraft  to  meet  all  its  needs.  Clearly  the  DoD  does  not  have  to  defeat 
the  adversary  with  a  single  system  and  should  make  its  acquisition  deci¬ 
sions  with  simplicity  in  mind. 

Complexity  is  the  antithesis  of  affordable,  on-time  systems.  Dan 
Ward,  in  particular,  has  been  writing  on  this  topic  for  some  time  (Ward, 
2012).  As  discussed  in  Department  of  Defense  Instruction  (DoDI) 
5000.02  and  the  Better  Buying  Power  initiatives  promulgated  by  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  Frank 
Kendall,  a  proven  successful  approach  has  been  taking  programs  in 
bite-sized  increments,  shooting  for  the  “80  percent  solution”  with  evo¬ 
lutionary  systems  to  follow.  It  may  be  far  better  to  take  a  less  capable, 
mature  system  now  and  build  up  to  the  full  capability  through  evolution¬ 
ary  or  block  development.  The  F-16  is  a  notable  example  of  incremental 
development,  taking  the  “best”  that  could  be  made  quickly  for  a  reason¬ 
able  cost,  but  adjusting  to  new  technologies  and  adapting  for  the  current 
challenges  and  operational  experience  (Quadrennial  Defense  Review 
Independent  Panel,  2010).  This  reinforces  the  need  to  set  realistic  goals 
to  meet  the  schedule. 


Starting  more  programs  than  the  Service  can 
afford  creates  inefficiencies  by  lengthening 
programs.  Inevitably,  more  programs  will  be 
competing  for  the  same  limited  funds,  thereby 
creating  a  slow,  vicious  death  spiral  cycle  due  to 
sparse  budgets 


Initiate  Fewer  Programs 

Although  DoDI  5000.02  stipulates  full  funding  as  an  entrance 
requirement  for  a  development  effort’s  individual  phases,  the  DoD 
starts  more  programs  than  it  can  hope  to  fund  through  full  produc¬ 
tion  (Chaplain,  2011).  This  practice  creates  a  myriad  of  problems— it 
lengthens  acquisition  cycles,  creates  pressure  to  underestimate  costs, 
and  eventually  leads  to  quantity  cuts  that  can  precipitate  a  program’s 
death  spiral.  This  results  from  the  DoD  and  the  Services  pursuing 
perceived  higher  performance  through  a  more  robust  portfolio  of  capa¬ 
bilities.  However,  more  often  than  not,  this  impulse  is  driven  by  a  lack 
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of  clear  priorities  on  which  capabilities  or  systems  should  be  developed. 
Indeed,  the  need  for  better  prioritization  has  been  tacitly  acknowledged 
through  the  recent  revamping  of  the  Joint  Capabilities  Integration  and 
Development  System.  A  consistent  and  analytically  based  prioritization 
of  systems  that  can  close  the  maximum  number  of  capability  gaps  could 
encourage  a  reduction  of  the  chaotic  competition  that  plagues  the  defense 
acquisition  system  right  now.  For  example,  the  DoD  could  adopt  an  algo¬ 
rithm  similar  to  one  RAND  developed  for  prioritizing  systems  early  in 
the  acquisition  cycle  (Chow,  Silberglitt,  &  Hiromoto,  2009). 

Starting  more  programs  than  the  Service  can  afford  creates  inef¬ 
ficiencies  by  lengthening  programs.  Inevitably,  more  programs  will  be 
competing  for  the  same  limited  funds,  thereby  creating  a  slow,  vicious 
death  spiral  cycle  due  to  sparse  budgets.  Eventually,  the  DoD  needs  to 
admit  some  of  the  future  systems  are  just  not  going  to  happen  without 
huge  infusions  of  cash,  which  seems  unlikely.  Banking  on  sufficient  funds 
to  cover  all  programs  through  full  production  places  other  systems  in 
jeopardy,  and  while  the  investments  may  not  be  wasted  completely,  the 
money  could  be  better  spent  elsewhere  to  produce  on-time  programs. 

Conclusions 

Previous  research  has  failed  to  develop  a  theoretical  frame¬ 
work  from  which  to  analyze  tradeoffs  within  the  acquisition  system. 
Although  the  acquisition  community  has  produced  many  potentially 
useful  observations  and  recommendations  since  the  early  1990s,  this  gap 
in  the  literature  has  arguably  marginalized  efforts  to  implement  worth¬ 
while  policy  changes.  By  making  it  seem  as  if  the  recommended  policy 
changes  were  not  motivated  by  any  single  overarching  guiding  principle, 
the  impetus  for  a  paradigm  shift  was  weakened,  and  piecemeal  changes 
and  stalls  were  the  result.  This  research  effort  seeks  to  unite  many  of  the 
previous  recommendations  under  a  single  theoretical  rubric. 

While  this  article  has  taken  on  a  distinctly  mathematical  tone,  the 
primary  objective  has  been  to  develop  an  internally  consistent  frame¬ 
work  for  the  values  DoD  leaders  should  communicate  to  the  acquisition 
community.  That  is,  the  complexity  of  the  defense  acquisition  sys¬ 
tem  precludes  resolution  of  priorities  through  a  simple  mathematical 
programming  problem,  but  the  insight  gained  through  the  inherent 
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prioritization  represented  in  the  model  allows  for  clarity  of  purpose.  For 
the  process  to  be  disciplined,  senior  leaders  must  transmit  unambiguous 
values  through  their  statements  and  actions. 

In  keeping  with  this  framework,  the  popular  DoD  convention  of 
focusing  primarily  on  costs  should  be  eschewed,  and  more  attention 
should  be  paid  to  meeting  the  schedule.  Empirical  results  have  demon¬ 
strated  that  cost  performance  is  likely  to  improve  through  adoption  of  a 
new  mindset,  and  the  more  realistic  expectations  concerning  a  system’s 
performance  that  are  implied  by  this  conceptual  model  will  undoubtedly 
improve  both  cost  and  schedule  outcomes. 
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Review: 

Innovation  during  the  period  between  World  Wars  I  and  II?  How 
could  that  be?  The  Nation  could  little  afford  to  build  new  warships. 
Treaties  limited  the  number,  type,  and  size  of  capital  warships  as  well  as 
fortifications  in  the  Western  Pacific.  Yet,  the  Navy  knew  it  had  to  imple¬ 
ment  new,  emerging  technologies  such  as  naval  aviation  and  undersea 
warfare.  Navy  leaders  recognized  they  had  to  look  for  innovative  ways 
to  overcome  the  decreasing  strength  of  their  fleet  relative  to  Japan’s.  As 
Professor  John  Kuehn  emphasizes,  this  multidimensional  threat  drasti¬ 
cally  altered  the  way  the  Navy  viewed  the  application  of  sea  power.  The 
simple  premise  of  this  brilliant  book  is  “the  U.S.  Navy’s  contributions 
to  victory  in  the  Pacific. ..can  be  understood  only  by  studying  how  the 
General  Board. ..constructed  the  ‘treaty  navy’  during  the  period  between 
the  wars.” 

The  General  Board  was  established  as  an  advisory  body  by  the 
Secretary  of  the  Navy  in  1901.  Its  members  were  senior-  and  mid-level 
officers  with  proven  experience  and  promise.  The  Board  hastened 
collaboration  between  the  Naval  War  College,  the  Bureaus  (now  the 
Systems  Commands),  and  the  Chief  of  Naval  Operations.  It  held  iterative 
deliberations  concerning  naval  warfare  strategies,  new  technologies 
and  systems,  and  the  structure  and  size  of  the  U.  S.  Navy  Fleet.  The 
Board  collaborated  closely  with  the  Bureau  of  Construction  and  Repair 
(BuC&R)— now  the  Naval  Sea  Systems  Command  (NAVSEA)— tasking 
BuC&R  to  conduct  extensive  ship  design  studies  to  determine  the  size 
and  structure  of  an  affordable  fleet.  Although  the  Board’s  official  role  was 
advisory,  its  actual  influence  was  much  greater.  It  had  the  final  word  on 
ship  design  decisions,  including  critical  operational  requirements  and 
costs.  Professor  Kuehn  provides  a  captivating  description  of  how  the 
Navy  was  transformed  from  a  battleship-centric  Fleet  to  an  efficient 
treaty  Fleet,  designed  to  operate  at  extreme  distances  without  available 
bases,  that  by  1937  also  included  aircraft  carriers,  cruisers,  destroyers, 
submarines,  and  new  types  of  logistics  support  ships. 

All  defense  acquisition  professionals  should  study  this  excep¬ 
tional  book,  which  describes  the  elements  and  processes  for  successful 
acquisition  outcomes.  Professor  Kuehn  stresses  the  General  Board’s 
collaborative  process  demonstrates  that  innovation  can  occur  in  the  face 
of  constraints.  MIT’s  Eric  von  Hippel,  who  has  done  pioneering  research 
in  new  product  innovation,  emphasizes  that  one  of  the  most  important 
steps  to  innovative  concept  development  and  cutting  concept  develop¬ 
ment  time  and  cost  is  for  lead  users— users  like  senior  Fleet  operators  at 
the  leading  edge  of  products— to  assess  their  own  needs  and  create  the 
design  concept  that  satisfies  their  own  needs.  His  research  validates 
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what  Professor  Kuehn  discovered:  that  there  are  very  few— maybe  even 
no— conditions  under  which  properly  equipped  users  engaged  in  open 
innovation  cannot  outdo  closed,  manufacturer-based  innovators.  This 
same  “open  innovation”  process  was  also  followed  by  successor  boards 
such  as  the  Ship  Characteristics  Improvement  Board  (SCIB)  during  the 
build-up  to  a  600-ship  Fleet  in  the  1980s  and  1990s.  Unfortunately,  the 
SCIB  was  abolished  around  2000  and  has  not  been  reconstituted.  The 
Performance  Assessment  and  Root  Cause  Analysis  (PARCA)  Office 
within  the  Department  of  Defense  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics,  recently  highlighted 
the  root  causes  for  major  defense  acquisition  programs  with  critical 
cost  growth  as  part  of  the  Nunn-McCurdy  breach  certification  process. 
PARCA  emphasized  that  unrealistic  estimates  are  generally  caused  by 
the  invalidity  of  major  assumptions  not  methodological  errors.  This 
has  led  to  what  PARCA  referred  to  as  “framing  assumptions”  early  in 
an  acquisition  program,  which  put  the  program  on  an  initial  path  for 
success  or  failure.  The  common  incorrect  framing  assumption  made 
by  acquisition  programs  with  critical  cost  growth  was  the  “Design  is 
mature.”  In  his  book,  Professor  Kuehn  has  captured  how  the  General 
Board  managed  technical  risks  to  ensure  a  mature  design  before  entering 
into  a  shipbuilding  contract.  Again,  this  is  a  must  read  for  you  “back-to- 
the-future”  types. 


Mr.  Robert  G.  “Bob”  Keane  is  currently  the  President  of  Ship  Design 
USA,  Inc.  Prior  to  starting  his  own  consulting  firm,  he  worked  at  the 
Naval  Sea  Systems  Command  for  35  years.  He  was  a  member  of  the 
Senior  Executive  Service  for  21  years  and  rose  to  the  senior  executive 
leadership  positions  of  Chief  Naval  Architect  of  the  Navy  and  the  Navy’s 
Chief  of  Ship  Design. 
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IN  GENERAL 


We  welcome  submissions  from  anyone  involved  in  the  defense  acqui¬ 
sition  process.  Defense  acquisition  is  defined  as  the  conceptualization, 
initiation,  design,  development,  testing,  contracting,  production,  deploy¬ 
ment,  logistics  support,  modification,  and  disposal  of  weapons  and  other 
systems,  supplies,  or  services  needed  for  a  nation’s  defense  and  security, 
or  intended  for  use  to  support  military  missions. 

Research  involves  the  creation  of  new  knowledge.  This  generally 
requires  using  material  from  primary  sources,  including  program  docu¬ 
ments,  policy  papers,  memoranda,  surveys,  interviews,  etc.  Articles  are 
characterized  by  a  systematic  inquiry  into  a  subject  to  discover/revise 
facts  or  theories  with  the  possibility  of  influencing  the  development  of 
acquisition  policy  and/or  process. 

We  encourage  prospective  writers  to  coauthor,  adding  depth  to 
manuscripts.  It  is  recommended  that  a  mentor  be  selected  who  has 
been  previously  published  or  has  expertise  in  the  manuscript’s  subject. 
Authors  should  be  familiar  with  the  style  and  format  of  previous  Defense 
ARJs  and  adhere  to  the  use  of  endnotes  versus  footnotes,  formatting  of 
reference  lists,  and  the  use  of  designated  style  guides.  It  is  also  the  respon¬ 
sibility  of  the  corresponding  author  to  furnish  a  government  agency/ 
employer  clearance  with  each  submission. 


MANUSCRIPTS 


Manuscripts  should  reflect  research  of  empirically  supported  experi¬ 
ence  in  one  or  more  of  the  areas  of  acquisition  discussed  above. 
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Research  articles  may  be  published  both  in  print  and  online,  or 
as  a  Web-only  version.  Articles  that  are  4,500  words  or  less  (excluding 
abstracts,  references,  tables,  and  figures)  will  be  considered  for  both  print 
as  well  as  Web  publication.  Articles  between  4,500  and  10,000  words  will 
be  considered  for  Web-only  publication,  with  an  abstract  included  in  the 
print  version  of  the  Defense  ARJ.  In  no  case  should  article  submissions 
exceed  10,000  words. 

Audience  and  Writing  Style 

The  readers  of  the  Defense  ARJ  are  primarily  practitioners  within 
the  defense  acquisition  community.  Authors  should  therefore  strive  to 
demonstrate,  clearly  and  concisely,  how  their  work  affects  this  commu¬ 
nity.  At  the  same  time,  do  not  take  an  overly  scholarly  approach  in  either 
content  or  language. 

Format 

Please  submit  your  manuscript  with  references  in  APA  format  (author- 
date-page  number  form  of  citation)  as  outlined  in  the  Publication  Manual 
of  the  American  Psychological  Association  (6th  Edition).  For  all  other  style 
questions,  please  refer  to  the  Chicago  Manual  of  Style  (15th  Edition). 

Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librar¬ 
ian  in  completing  citation  of  government  documents  because  standard 
formulas  of  citations  may  provide  incomplete  information  in  reference 
to  government  works.  Helpful  guidance  is  also  available  in  Garner,  D. 
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L.,  and  Smith,  D.  H.,  1993,  The  Complete  Guide  to  Citing  Government 
Documents:  A  Manual  for  Writers  and  Librarians  (Rev.  Ed.),  Bethesda, 
MD:  Congressional  Information  Service. 

Pages  should  be  double-spaced  and  organized  in  the  following  order: 
title  page,  abstract  (120  words  or  less),  two-line  summary,  list  of  key¬ 
words  (five  words  or  less),  body  of  the  paper,  reference  list  (works  cited), 
author’s  note  (if  any),  and  any  figures  or  tables. 

Figures  or  tables  should  not  be  inserted  (or  embedded,  etc.)  into  the 
text,  but  segregated  (one  to  a  page)  following  the  text.  When  material  is 
submitted  electronically,  each  figure  or  table  should  be  saved  to  a  sepa¬ 
rate,  exportable  file  (i.e.,  a  readable  EPS  file).  For  additional  information 
on  the  preparation  of  figures  or  tables,  see  CBE  Scientific  Illustration 
Committee,  1988,  Illustrating  Science:  Standards  for  Publication, 
Bethesda,  MD:  Council  of  Biology  Editors.  Restructure  briefing  charts 
and  slides  to  look  similar  to  those  in  previous  issues  of  the  Defense  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authors) 
should  attach  to  the  manuscript  a  signed  cover  letter  that  provides  all  of 
the  authors’  names,  mailing  and  e-mail  addresses,  as  well  as  telephone 
and  fax  numbers.  The  letter  should  verify  that  the  submission  is  an 
original  product  of  the  author;  that  it  has  not  been  previously  published 
in  another  journal  (monographs  and  conference  proceedings,  however, 
are  okay);  and  that  it  is  not  under  consideration  by  another  journal  for 
publication.  Details  about  the  manuscript  should  also  be  included  in 
this  letter:  for  example,  title,  word  length,  a  description  of  the  computer 
application  programs,  and  file  names  used  on  enclosed  CDs,  e-mail 
attachments,  or  other  electronic  media. 

COPYRIGHT 

The  Defense  ARJ  is  a  publication  of  the  United  States  Government 
and  as  such  is  not  copyrighted.  Because  the  Defense  ARJ  is  posted  as 
a  complete  document  on  the  DAU  homepage,  we  will  not  accept  copy¬ 
righted  manuscripts  that  require  special  posting  requirements  or 
restrictions.  If  we  do  publish  your  copyrighted  article,  we  will  print  only 
the  usual  caveats.  The  work  of  federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copyright  except  in  rare  cases. 
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Web-only  publications  will  be  held  to  the  same  high  standards  and 
scrutiny  as  articles  that  appear  in  the  printed  version  of  the  journal  and 
will  be  posted  to  the  DAU  Web  site  at  www.dau.mil. 

In  citing  the  work  of  others,  please  be  precise  when  following  the 
author-date-page  number  format.  It  is  the  contributor’s  responsibility  to 
obtain  permission  from  a  copyright  holder  if  the  proposed  use  exceeds 
the  fair  use  provisions  of  the  law  (see  U.S.  Government  Printing  Office, 
1994,  Circular  92:  Copyright  Law  of  the  United  States  of  America,  p.  15, 
Washington,  DC.).  Contributors  will  be  required  to  submit  a  copy  of  the 
writer’s  permission  to  the  Managing  Editor  before  publication. 

Policy 

We  reserve  the  right  to  decline  any  article  that  fails  to  meet  the 
following  copyright  requirements: 

•  The  author  cannot  obtain  permission  to  use  previously 
copyrighted  material  (e.g.,  graphs  or  illustrations)  in  the 
article. 

•  The  author  will  not  allow  DAU  to  post  the  article  in  our 
Defense  ARJ  issue  on  our  Internet  homepage. 

•  The  author  requires  that  usual  copyright  notices  be  posted 
with  the  article. 

•  To  publish  the  article  requires  copyright  payment  by  the 
DAU  Press. 


SUBMISSION 

All  manuscript  submissions  should  include  the  following: 

•  Cover  letter 

•  Author  checklist 

•  Biographical  sketch  for  each  author  (70  words  or  less) 
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•  Headshot  for  each  author  should  be  saved  at  no  less  than 
300  dpi  (dots  per  inch)  or  as  a  high-print  quality  JPEG  or 
Tiff  file  saved  as  no  less  than  5x7.  Please  note:  images  from 
Web,  PowerPoint,  or  e-mail  will  not  be  accepted  due  to  low 
image  quality. 

•  One  copy  of  the  typed  manuscript,  including: 

°  Title  (12  words  or  less) 

°  Abstract  of  article  (120  words  or  less) 

°  Two-line  summary 
°  Keywords  (5  words  or  less) 

°  Document  excluding  abstract  and  references  (4,500 
words  or  less  for  the  printed  edition  and  10,000  words 
or  less  for  online-only  content) 

These  items  should  be  sent  electronically,  as  appropriately  labeled 
files,  to  Defense  ARJ  Managing  Editor,  Norene  Fagan-Blanch  at:  Norene. 
Fagan-Blanch@dau.mil. 
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The  Defense  Acquisition  Research  Journal  (ARJ)  is  published 
at  the  Defense  Acquisition  University  (DAU),  Fort  Belvoir,  VA.  The 
University  publishes  four  issues  annually.  The  Director  of  the  Defense 
Acquisition  University  (DAU)  Press  is  Randy  Weekes,  the  Managing 
Editor  of  the  Defense  ARJ  is  Norene  Fagan-Blanch,  and  the  publisher  is 
the  DAU  Press.  All  are  collocated  at  the  following  address: 

Defense  Acquisition  University 
Attn  DAU  Press  (Defense  ARJ) 

9820  Belvoir  Road,  Suite  3 
Fort  Belvoir,  VA  22060-5565 

Average  Number  of  Copies  Each  Issue  During  the  Preceding  12  Months: 

A.  Total  number  of  copies  printed  (net  press  run) . 3,754 

B.  Paid  and/or  requested  circulation: 

1.  Sales  through  dealers  and  carriers,  street  vendors,  and  counter  sales . 0 

2.  Mail  subscriptions  paid  and/or  requested . 3,379 

C.  Total  paid  and/or  requested  circulation . 3,379 

D.  Free  distribution  by  mail,  carrier,  or  other  means;  samples, 

complementary,  and  other  free  copies . 112 

E.  Total  distribution . 3,566 

F.  Copies  not  distributed: 

1.  Office  use,  leftover,  unaccounted,  spoiled  after  printing . 188 

2.  Returns  from  news  agents . 0 

G.  Total . 3,754 

Actual  Number  of  Copies  of  Single  Issue  Published  Nearest  to  Filing  Date: 

A.  Total  number  of  copies  printed  (net  press  run) . 3,626 

B.  Paid  and/or  requested  circulation: 

1.  Sales  through  dealers  and  carriers,  street  vendors,  and  counter  sales . 0 

2.  Mail  subscriptions  paid  and/or  requested . 3,267 

C.  Total  paid  and/or  requested  circulation . 3,267 

D.  Free  distribution  by  mail,  carrier,  or  other  means;  samples, 

complementary,  and  other  free  copies . 137 

E.  Total  distribution . 3,585 

F.  Copies  not  distributed: 

1.  Office  use,  leftover,  unaccounted,  spoiled  after  printing . 41 

2.  Returns  from  news  agents . 0 

G.  Total . 3,626 
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Publication  Date 
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been  received  within  48  hours  of  its  arrival.  Following  an  initial  review, 
submissions  will  be  referred  to  referees  and  for  subsequent  consideration 
by  the  Executive  Editor,  Defense  ARJ. 


Contributors  may  direct  their  questions  to  the  Managing  Editor, 
Defense  ARJ,  at  the  address  shown  below,  or  by  calling  703-805-3801 
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